In the second tech blog in this miniseries, we discussed neural networks, and how this form of machine learning can be used to build a sound classifier. In this miniserie, we will cover the digitalization of audio.
With audio input and corresponding annotations as “ground truth”, a neural network can be trained to assign labels to audio accurately. However, it is important to realize that computer algorithms never have the original, analog sound waves as input; it has to be some sort of digital representation of the sound. This means that the sound waves need to be picked up by a microphone first, which transfers sound waves into an electrical signal, which is subsequently turned into a digital signal by an Analog/Digital converter. This conversion is done by sampling, which basically means that the converter takes “samples” of its analog input at a very high rate. For example, for CD quality audio, this sampling happens at a rate of 44.1 kHz, meaning 44,100 times per second. “High resolution” audio exists with higher sampling rates, like 96 kHz, 192 kHz, or even higher in some cases.
When a digitally sampled audio signal is played back, it is converted back to an electrical signal by a D/A converter, which is then transformed back into sound waves by loudspeakers. On paper, D/A conversion is able to reconstruct the original signal back to a maximum of half the sample rate. This means that if a signal was sampled at, say, 48 kHz, the original signal can be stored and reconstructed in the 0 to 24 kHz range. This upper limit is called the “Nyquist frequency”. Keep in mind that the human hearing is able to hear roughly between 20 and 20 kHz, so this range should be more than sufficient. However, one consequence of sampling is that frequencies above the Nyquist frequency should be filtered out before A/D conversion, to avoid an unwanted effect called “aliasing”, where these higher frequencies will be misinterpreted by the converter and become audible distortion artifacts at lower frequencies. This is why, in practice, low-pass filters are put in place in front of A/D converters to filter out these high frequencies first. Such a filter is called an “anti-aliasing” filter. These filters are never perfect, unfortunately, and do have a specific “bandwidth” themselves. When the sample rate is too low, there is the danger this filtering becomes audible. This is why professional recording studios, for example, prefer working with sample rates higher than 44.1 kHz.
Besides the sample rate, there is another aspect of A/D conversion which makes the digitally sampled signal less than perfect: bit depth, which describes how many bits are used to store each sample. For CD-quality audio, each sample is stored using 16 bits, which means that the “amplitude” of the analog signal is transformed (or “quantized”) into 2^16 = 65536 discrete steps. As a result, the D/A converter will be able to reproduce the original, analog amplitude perfectly, but with a tiny bit of extra “quantization noise”. The more bits that were used when storing the signal, the lower the level of this extra noise is. So, in a way, the bit depth determines the difference between the highest level and the lowest level of sound that can be stored. This is also called the “dynamic range”, which is roughly 96 dB (decibels) for a 16 bit signal (the exact number depends on optional tricks being applied, like “dithering”, which should be a topic for another time).
Both the sample rate and the bit depth of the digital audio signal are very important for sound classification. As explained, the sample rate determines the highest sound frequency that can be detected accurately, which may become very important for some types of sounds like high-pitched alarms, or even the human voice. Studies found that for voice sometimes even energy above 8 kHz may contain important cues for humans. The dynamic range is very important as well: when the bit depth is too low, low-level sounds may become “buried” in the quantization noise, or high-level sounds may result in an effect called “clipping”, where the high peaks of the sound waves are literally “cut off” because there are no more bits left in the conversion.
So, low sample rates and low bit depths will lead to information loss, making it more difficult for the classifier to label the sound accurately. For Sound Intelligence’s classifiers, the input audio should have a sample rate of at least 16 kHz, and a bit depth of 16 bits per sample. Luckily, almost every device in today’s market is able to meet these requirements.
Finally, there are more factors in play that affect the quality of the audio input. For example, the microphone used has a “frequency response” (or “transfer response”) of its own. A perfect microphone will transfer each frequency equally (which is called a “flat response”), but no microphone is perfect. This means that some frequencies will be recorded louder than others. Even the room the sound was recorded in has a frequency response, because the surfaces present in the room will reflect and absorb the various frequencies differently. All frequency responses combined will have a significant effect on the recorded audio, and thus on sound event detection.
That’s it so far for an explanation of digital audio. In the next blog we will explain how the raw audio can be turned into something more useful for an audio classifier.
