Now that we discussed the digital audio input in the previous tech blog (link) in this miniseries, it is time to take a look at how we turn this input into something a machine learning model can work with.
As discussed in the second blog, a neural network (or any other machine learning model) needs an input in order to generate an output. This input can basically be anything, so nothing prevents us from using the raw, digitized microphone signal as input for the network. In fact, in some cases the raw digital signal is actually used as an input. For example, machine learning algorithms used to generate audio (e.g. AI-generated speech or music) are trained on raw audio signals. But, even for classification purposes training this way can be a good choice in some cases.
*Figure 1: A raw digital waveform of a speech signal. The waveform shows the signal’s amplitude as a function of time.*
However, in practice it turns out most machine learning algorithms benefit from a little bit of “help” at the input. In most applications, the raw amplitude is not enough for accurate results; we actually need to split the signal in different frequencies first and provide the algorithm the per-frequency (band) amplitude over time. One way to do this is by creating a so-called “spectrogram”, which shows the per-frequency energy over time:
*Figure 2: A spectrogram of the raw input signal showed in Fig. 1. This spectrogram shows the per-frequency amplitude as a function of time. The red parts have more energy than the blue parts.*
The spectrogram clearly shows interesting features about the speech signal that are not visible in the raw signal. For example, we clearly see the harmonics (“overtones”) of the speech signal as lines with more energy (red). In theory, a neural network could learn to analyze the per-frequency energy as well if it only gets the raw signal as input. But, this would require quite a lot of effort by the algorithm that could also be spend on other things (like classification). This is why we can “help” the algorithm by providing the spectrogram as input, which indeed yields better performing algorithms in almost all cases.
The example spectrogram plot also clearly shows that most of the energy of the signal is concentrated in the lower frequencies, roughly below 2000 Hz. In fact, almost all signals we encounter in reality show more energy in lower frequency bands compared to the higher ones, and our ears have adapted to this. The basilar membrane in our inner ear, which basically performs the splitting of energies over frequencies for us, has logarithmic properties: lower frequencies excite a bigger part of this membrane than the higher ones. This way, we end up with more resolution in the part of the signal that contains the most useful information.
For this reason, it would make sense to do something similar at the input of a machine learning algorithm. A popular method is to transform the spectrogram in a “mel spectrogram”, where the frequency bands are grouped into bands according to the “mel” scale. “mel” stands for “melody”, and this scale is designed such that humans perceive the distance between bands equally in terms of pitch. For example, we could transform the spectrogram from Fig. 2 into a mel spectrogram and end up with something like this:
*Figure 3: A mel spectrogram obtained from the spectrogram in Fig. 2, which shows the energy per frequency band over time, following a mel scale.*
The interesting thing about this mel spectrogram is that we see more detail now in the lower part, even though the overall resolution is lower (multiple “raw” frequencies are “grouped” into the same mel band). This is a very useful trade-off for a machine learning algorithm: the input becomes smaller in terms of data points, while retaining all the useful information necessary for the task at hand. In fact, in turns out mel spectrograms are even useful for partially removing unwanted parts of the signal like noise. These features are the main reason almost all audio-related, non-generative machine learning algorithms use some form of mel spectrogram at the input.
We can take “helping the algorithm” even multiple steps further by adding more input features besides the mel spectrogram. Examples of such features are (root) pitch, spectral bandwidth, rhythm-based properties etc. Another type of processing we usually perform has to do with “normalization”. For example, we want to normalize the energy of the input, because we want a classifier to perform well regardless if an acoustic source is close to or at a larger distance from the microphone.
In the end, it is all a matter of finding input features that help the algorithm perform better, without making it too big or increasing the risk of over-training, which was discussed in an earlier blog in this mini-series.
In the next blog, we will discuss how we can actually form a useful dataset to train an algorithm on using the features discussed here.
Cover Image Credits: Mahdis Mousavi for Unsplash
