## Let's take a step back

Last time I said I had to try Autocorrelation in order to guess a note from an audio input. To explain this principle, I need to come back on a few things. I forgot to mention that we can represent frequencies as sine waves. For instance, here is how we could represent a frequency of 440Hz :

But most of the time, a sound is a sum of frequencies, for instance if we combine A4 (440Hz) and C#5 (554.37Hz), it looks like this :

But in real life, what we get is the sum of those frequencies, which looks like this :

As notes are louder or quieter, their frequency amplitudes differ. For instance, if A has 4 times more amplitude than C#, it results in a sine wave looking like this :

As you can see, a frequency is a pattern replicated within each period. Previously we relied on `getByteFrequencyData`

from the analyser to get frequencies in bins of `1.46484375Hz`

. But this led us to nothing. What we will do now, is access the sound information at a given time and verify if it repeats within a period. With this period we will then be able to calculate our main frequency : `frequency = 1/period`

.

Remember when I said that frequencies are sine waves ? This is where is comes handy. By offsetting it gradually we can verify if it repeats.

**Here is a proof of concept :**

Use the range input below to change the period of the red sine wave.

P = 0.00002s

F = 1/P = 50000Hz

You will probably find that the frequency is around 450Hz, it's due to rendering but anyway, it's close to our 440Hz, so, you get the idea !

## Let's build the tuner

First, I prefer to extract the logic to create our analyser inside a hook and use it on demand :

```
export const FFT_SIZE = 32768
export function useAnalyser(stream: MediaStream, fftSize: number = FFT_SIZE): [] | [AnalyserNode, MediaStreamAudioSourceNode] {
if (!stream ) {
return []
}
const ctx = new AudioContext()
const analyser = ctx.createAnalyser()
analyser.fftSize = fftSize
analyser.minDecibels = -100;
analyser.maxDecibels = -10;
analyser.smoothingTimeConstant = 0.85;
const sourceNode = ctx.createMediaStreamSource(stream)
sourceNode.connect(analyser)
return [analyser, sourceNode]
}
```

Then invoke the hook and create a function to start using our tuner extract timebased data from our analyser. It will use a setInterval that we can stop when we later want the tuner to stop.

```
const [analyser] = useAnalyser(audioStream, 2048)
function start() {
if (analyser) {
const interval = setInterval(() => {
const buffer = new Float32Array(analyser.fftSize)
analyser.getFloatTimeDomainData(buffer)
// This will be where we run our autoCorrelate function to guess the frequency
}, 500) as unknown as number
setGuessingInterval(interval)
}
}
```

We then use this wonderful function made for us by a PhD student, rewritten for reading purposes :

```
// Thanks to
// https://github.com/cwilso/PitchDetect/pull/23
// https://github.com/dalatant
// https://alexanderell.is/posts/tuner/
// https://en.wikipedia.org/wiki/Root_mean_square
function getRootMeanSquare(buffer: Float32Array) {
let sum = 0
for (let i = 0; i < buffer.length; i++) {
sum += buffer[i] * buffer[i]
}
return Math.sqrt(sum / buffer.length)
}
function buildBufferWithThreshold(buffer: Float32Array) {
// Since the main sound is a sum of frequencies it can be seen as a composed sine wave
// We can predict that it will repeat, so we can narrow the buffer down, to several occurences with a threshold
let r1 = 0;
let r2 = buffer.length - 1;
let thres = 0.2;
// Split buffer un half and check where we lose frequency amplitude so we can work on less data
for (let i = 0; i < buffer.length / 2; i++) {
if (Math.abs(buffer[i]) < thres) {
r1 = i;
break;
}
}
for (let i = 1; i < buffer.length / 2; i++) {
if (Math.abs(buffer[buffer.length - i]) < thres) {
r2 = buffer.length - i;
break;
}
}
return buffer.slice(r1, r2)
}
// Implements the ACF2+ algorithm
export function autoCorrelate(buffer: Float32Array, sampleRate: number) {
// Calculate the root mean square to see if we have enough signal to perform operations
let rootMeanSquare = getRootMeanSquare(buffer)
if (rootMeanSquare < 0.01) {
return -1;
}
const newBuffer = buildBufferWithThreshold(buffer)
// Build correlation array containing ordinates values multiplied by a moving offset.
// The index that will have the greater value will be the peak of our sine wave
let correlation = new Array(newBuffer.length).fill(0);
for (let i = 0; i < newBuffer.length; i++) {
for (let j = 0; j < newBuffer.length-i; j++) {
correlation[i] += newBuffer[j] * newBuffer[j+i];
}
}
let d = 0;
// Find the first bottom within our new range
while (correlation[d] > correlation[d + 1]) {
d++;
}
let maxVal = -1
let maxPos = -1;
// From that index find the first peak within the correlation array, our desired abscissa
for (var i = d; i < newBuffer.length; i++) {
if (correlation[i] > maxVal) {
maxVal = correlation[i];
maxPos = i;
}
}
var T0 = maxPos;
// From the original author:
// interpolation is parabolic interpolation. It helps with precision. We suppose that a parabola pass through the
// three points that comprise the peak. 'a' and 'b' are the unknowns from the linear equation system and b/(2a) is
// the "error" in the abscissa. Well x1,x2,x3 should be y1,y2,y3 because they are the ordinates.
// ----
// Note: x1, x2 and x3 were the previous names of y1, y2 and y3
// ----
// From me:
// What I can comprehend on this, is that we have a parabolic equation, but are unsure which of those three points
// is the closest to our desired frequency. This bits sorts it out by comparing which is closer to the parabolic
// peak.
let y1 = correlation[T0 - 1]
let y2 = correlation[T0]
let y3 = correlation[T0 + 1];
let a = (y1 + y3 - 2 * y2) / 2;
let b = (y3 - y1) / 2;
if (a) {
T0 = T0 - b / (2 * a);
}
return sampleRate / T0;
}
```

It may seem complicated but it only is what I was writing about before. Calculate the autocorrelation, finding it's first peak, and boom, we have our **period**. We then just add it to our `start`

function :

```
function start() {
if (analyser) {
const interval = setInterval(() => {
const buffer = new Float32Array(analyser.fftSize)
analyser.getFloatTimeDomainData(buffer)
// Guess the frequency
setGuessedFrequency(autoCorrelate(buffer, analyser.context.sampleRate))
}, 500) as unknown as number
setGuessingInterval(interval)
}
}
```

Let's just add a function to stop the tuner, and we'll be ready to wrap it up !

```
function stop() {
clearInterval(guessingInterval)
}
```

That's it, we have a tuner, that every 500ms will take the sound input and guess its main pitch.

## The process explained with charts

At a given time, we extract our timebased values, notice how there is 2048 values, it's our FFT_SIZE :

We filter values that are above a specific threshold. Well, it doesn't do that much in our case, but anyway :

We calculate correlation for each abscissa :

And we find our period :

We have a period of 401, but our sample rate is 44100 samples per second. What remains is dividing our sample rate by our period, to obtain our desired frequency : `44100 / 401 = 109.975062344Hz`

. Which is really close to the frequency of an A string on a guitar : `110.0Hz`

! This is the string I plucked for the test. It means that it works, and I am in tune, just barely barely flat !

## What if we want to use it to tune a guitar ?

Well, that was the purpose of this whole experiment. Let's use MTTS to generate an array containing tuning notes for our guitar. It is a library that I created to manipulate music theory conceps in typescript. We can use it to easily create notes, scales, chords, etc. :

```
const notes = [
Note.fromSPN('E2'),
Note.fromSPN('A2'),
Note.fromSPN('D3'),
Note.fromSPN('G3'),
Note.fromSPN('B3'),
Note.fromSPN('E4'),
]
```

We now have an array containing notes to test our guesssed frequency against. Let's create a function that recursively take a frequency and an array of notes and returns the note with the closest frequency.

```
function findClosestNote(searchedFrequency: number, notesInRange: Note[]): Note {
if (notesInRange.length <= 3) {
let noteFound = notesInRange[0]
let closest = Math.abs(searchedFrequency - noteFound.frequency)
for (let i = 1; i < notesInRange.length; i++) {
const frequencyDifference = Math.abs(searchedFrequency - notesInRange[i].frequency)
if (closest > frequencyDifference) {
noteFound = notesInRange[i]
closest = frequencyDifference
}
}
return noteFound
}
const secondHalf = notesInRange.slice(notesInRange.length / 2, notesInRange.length)
if (searchedFrequency < secondHalf[0].frequency) {
return findClosestNote(searchedFrequency, notesInRange.slice(0, (notesInRange.length / 2) + 1))
}
return findClosestNote(searchedFrequency, notesInRange.slice((notesInRange.length / 2) - 1, notesInRange.length))
}
```

And use it on our frequency inside our component :

```
useEffect(() => {
if (guessedFrequency > 0) {
setGuessedNote(findClosestNote(guessedFrequency, notes))
} else {
setGuessedNote(undefined)
}
return undefined
}, [setGuessedNote, guessedFrequency])
```

Just to wrap it up, we add wheter our frequency is flat or sharp to our closest note. To avoid testing against a fixed frequency, that is impossible to have on a guitar, we add a margin of `0.3Hz`

.

```
const AVERAGE_FLUCTUATION = 0.3
const inTune = Math.abs(guessedFrequency - guessedNote?.frequency) < AVERAGE_FLUCTUATION
const sharp = Math.abs(guessedFrequency - guessedNote?.frequency) > AVERAGE_FLUCTUATION && (guessedFrequency - guessedNote?.frequency) > 0
const flat = Math.abs(guessedFrequency - guessedNote?.frequency) > AVERAGE_FLUCTUATION && (guessedFrequency - guessedNote?.frequency) < 0
```

And here we have our tuner :

The full code of this project can be found here : https://github.com/FaXaq/website/tree/master/app/blog/tuner-pt2