Docs / Audio Processing / Acoustic Echo Cancellation (AEC)

Acoustic Echo Cancellation (AEC)

When a device listens and talks at the same time, the sound leaving the speaker feeds back into the microphone that is listening for input. Acoustic echo cancellation removes that output from the capture.

The problem

Whatever leaves the speaker travels back to the microphone a few milliseconds later, mixed in with whatever the person is actually saying. The microphone cannot tell the two apart, so everything downstream treats your own output as though someone in the room had said it.

In practice that means:

AEC is built into Decibri. It takes the audio going to the speaker and removes that sound from the microphone capture. Switching it on is one option on the microphone, plus one call each time a block of audio goes to the speaker.

When to use AEC

Use it when playback and capture overlap in time and the sound is leaving through a loudspeaker. The test is simply whether the two are ever live at the same moment.

Common cases: an assistant you can interrupt mid-sentence, a conferencing client on laptop speakers, a speakerphone, a kiosk, or a robot that talks to you while still listening for your next instruction.

An agent you can interrupt is the clearest case. It is only pleasant to talk to if you can cut in while it is still speaking, and that means leaving the microphone open through its own reply. Without cancellation, the first thing it hears when it starts speaking is itself.

When not to use AEC

Where it sits in the chain

Capture is normalised to mono at the sample rate you asked for, and cancellation runs on that, ahead of every ACE conditioning stage. The speech detector reads the signal the moment cancellation has finished with it.

capture → echo cancellation → ACE conditioning

That ordering has two consequences.

The detector reads after cancellation. With echo cancellation on, vadScore and the speech and silence events describe the person in the room, not your own playback. Read the limits section before relying on this, because it reduces false triggering rather than eliminating it.
Gain stages run after cancellation, never before. The canceller sees the capture at its natural level. If you set agc or limiter, they act on the already-cancelled signal. There is no ordering for you to configure.

Cancellation subtracts an estimate, and where the estimate is wrong the result can exceed the level of the original capture. With dtype: 'int16' the format clamps, so a sample arrives at full scale rather than wrapping. With float32 and no limiter, clamp in your own code or set limiter, which runs last and holds the output at its ceiling.

A working example

The simplest case: the audio is already prepared, and you are playing it through Decibri's own speaker. Feed each block to both the speaker and the canceller.

from decibri import File, Microphone, Speaker

RATE = 16000

# Whatever you are about to play. Any mono PCM at the capture rate works.
with File("speech.wav", sample_rate=RATE) as f:
    audio = b"".join(f)

block_bytes = RATE // 10 * 2  # 100 ms of int16

with Microphone(sample_rate=RATE, aec="tau") as mic, Speaker(sample_rate=RATE) as speaker:
    for i in range(0, len(audio), block_bytes):
        block = audio[i : i + block_bytes]
        speaker.write(block)             # play it
        mic.push_aec_reference(block)    # and tell the canceller you played it
        near = mic.read(timeout_ms=100)  # microphone audio, echo removed

That is the whole integration. One option on the microphone, one call per block of playback.

The reference signal

This is the part to get right. Everything else on this page is defaults.

What the reference is

The reference is the audio that went out to the speaker: the same samples you handed to the output device, at the moment you handed them over.

It is not the source material as it stood before you processed it. It is not the file as it sat on disk. It is not what the microphone picked up. It is the output itself, as sent.

So if you change the audio on its way to the speaker, by adjusting the volume, resampling it, or mixing in a notification chime, hand over what came out of that step rather than what went into it. The canceller is searching your capture for this exact signal, and the closer it matches what actually left the speaker, the more it can remove.

The reference is always mono, and always in the same sample format as the microphone, int16 or float32 as your dtype says.

Silence counts

The reference is a continuous stream. Every moment of capture has a matching moment of playback, and while nothing is being played, that playback is silence.

You do not have to push that silence yourself. Decibri fills the gaps, so a caller that simply stops pushing between one utterance and the next has told the canceller that nothing played, which is true. The alignment survives the gap and carries on when you start pushing again.

Worth stating plainly, because the next rule depends on it: not pushing does not mean "nothing happened", it means "silence played".

Push in played order

Push blocks in the order they were played, with nothing left out. Decibri lays the reference down against the capture in the order it arrives, so a block you skip is not treated as a gap. It is treated as silence, and everything you push after it lands in the wrong place.

If an error path drops a block, or an async boundary reorders them, push what actually played in the order it actually played. If you truly cannot, pushing nothing is better than pushing something out of order.

Push in step with playback, block by block, rather than handing over a long buffer in advance. The reference is held in a queue that spans two seconds at the reference rate; a single push longer than that loses the excess, counted in referenceDropped.

Matching the playback rate

This is the one that fails silently. Playback and capture rates commonly differ. Speech synthesis services typically return 24 kHz audio, while a capture feeding a speech recogniser usually runs at 16 kHz. A browser or conferencing stack will hand you 48 kHz.

A mismatched rate raises nothing. If the reference arrives at a rate other than the capture rate and you have not said so, Decibri treats the samples as if they were at the capture rate. The timing is wrong, the canceller never finds the echo, and nothing reports an error. Capture keeps running, audio keeps flowing, no exception is raised, and the echo is still there.

Declare the reference rate and Decibri converts it for you:

Microphone(
    sample_rate=16000,
    aec=Aec(model="tau", reference_sample_rate=24000),
)

Then push the audio at its own rate, unconverted. Do not resample it yourself first; if you do, do not declare the rate, because the two together apply the conversion twice.

The check that catches this: with playback running, read the metrics. If delaySamples is null after a few seconds of far-end audio, the rate is the first thing to look at.

When something else plays the audio

Decibri does not have to be the thing playing. A speech synthesis service, a conferencing track, a browser player, a game engine: any of them can do the playing and cancellation still works. What matters is that your code handles the samples on their way out. Wherever it hands them to the player is where you push, with the same buffer and the rate declared.

from decibri import Aec, File, Microphone, Speaker

CAPTURE_RATE = 16000
PLAYBACK_RATE = 24000  # whatever the player runs at

# Audio from somewhere else, at that source's own rate.
with File("speech.wav", sample_rate=PLAYBACK_RATE) as f:
    audio = b"".join(f)

mic = Microphone(
    sample_rate=CAPTURE_RATE,
    aec=Aec(model="tau", reference_sample_rate=PLAYBACK_RATE),
)
player = Speaker(sample_rate=PLAYBACK_RATE)

block_bytes = PLAYBACK_RATE // 10 * 2  # 100 ms at the playback rate

with mic, player:
    for i in range(0, len(audio), block_bytes):
        block = audio[i : i + block_bytes]
        player.write(block)            # hand it to the player
        mic.push_aec_reference(block)  # and push the same bytes, unconverted
        near = mic.read(timeout_ms=100)

One thing this pattern does not cover: if the player converts the audio again before it reaches the speaker, push what you gave the player and accept that the reference is one step removed from what was actually played. It usually still works.

Push only while capture is running

A push before capture is running is discarded silently. It is not counted as a drop, and no error is raised.

The reliable check is the metrics accessor: it returns null until capture is running, and an object once it is. In Python, the context manager and start() both return with capture running, so a push straight afterwards is accepted. In Node the stream starts when you begin consuming it, so push from inside your data handler or your playback loop, not immediately after the constructor returns.

Configuration

aec accepts a model name for the defaults, or an object to tune it.

from decibri import Aec, Microphone

# Short form: the model with its defaults.
a = Microphone(sample_rate=16000, aec="tau")

# Object form: the same model, every option set to its default.
b = Microphone(
    sample_rate=16000,
    aec=Aec(
        model="tau",
        tail_ms=200,
        suppression="conservative",
        reference_sample_rate=16000,
    ),
)

model

'tau' is the model available today. It carries no model file and needs no download. Passing the short form and passing { model: 'tau' } are the same thing.

An unrecognised name is rejected when the microphone is constructed, with a message naming the accepted set. Leave this alone unless a future release adds another name.

tailMs / tail_ms

How much echo delay spread the canceller can account for, in milliseconds. Accepted range 16 to 500, default 200.

Raise it when the room is large or reverberant, or when the audio path adds delay, for example Bluetooth output or a device with deep buffering. Lower it for a small enclosure with a short direct path. Longer settings ask more of the machine, so prefer the default unless you have a reason.

suppression

'conservative' (the default) attenuates the residue the linear stage leaves behind. 'off' delivers the linear output as it stands.

Leave it at 'conservative' for a normal capture. Choose 'off' when downstream processing expects an unmodified near-end signal and you would rather handle the residue yourself.

referenceSampleRate / reference_sample_rate

The rate of the audio you push, in Hz. Accepted range 1000 to 384000. Unset means the reference is already at the capture rate.

Set it whenever your playback rate differs from your capture rate. See the rate section above; this is the option that exists for it.

Constraints on the capture

Echo cancellation requires sampleRate between 8000 and 48000, narrower than the range sampleRate otherwise accepts. A rate that is fine for a plain capture is rejected once echo cancellation is on.

Capture is mono, as it is for every decibri capture.

Echo cancellation is available on native capture only. The browser build does not take this option, because browser capture carries the platform's own echoCancellation constraint, which is on by default.

Diagnostics

aecMetrics() in Node, aec_metrics() in Python. Both return null / None when the aec option is unset or capture is not running, and an object otherwise.

Node Python Type Meaning
delaySamples delay_samples number or null Active alignment between reference and capture. null while searching.
erleDb erle_db number Smoothed estimate of how much echo is being removed, in dB.
doubleTalk double_talk boolean Whether the near-end talker is believed active. Adaptation is held while true.
referenceStarved reference_starved number Near-end samples with no far-end sample available while aligned.
acquisitionParked acquisition_parked number Near-end samples processed while no alignment was active.
referenceReanchors reference_reanchors number Times the alignment was rebuilt after a capture discontinuity.
referenceDropped reference_dropped number Far-end samples discarded because a push exceeded the queue, at the reference rate.
referenceSilence reference_silence number Far-end samples supplied as silence because none were pushed, at the capture rate.

Working out what is wrong

Read the metrics while far-end audio is actually playing, and give it a few seconds. Then work down this list. The first match is your answer.

A note on erleDb. It rises while echo is being removed, sits near zero before the filter settles, and dips while adaptation is held during double talk. It moves around continuously in normal operation. Use it to see change over time, not as a pass mark; there is no threshold that means "working". 
def diagnose(mic: Microphone) -> str:
    m = mic.aec_metrics()

    if m is None:
        return "echo cancellation is off, or capture is not running"
    if m.delay_samples is not None:
        return f"aligned at {m.delay_samples} samples, double talk {m.double_talk}"
    if m.reference_silence >= m.acquisition_parked:
        return "no reference is reaching the canceller"
    if m.reference_dropped > 0:
        return "pushes are being discarded: push smaller blocks, in step with playback"
    return "reference is arriving but nothing has aligned: check the rate"

Limits

Read this before you design around echo cancellation. These are properties of the problem, and knowing them will save you building on an assumption that does not hold.

How much echo is removed depends on the room

Echo reduction is governed by the acoustic path between your speaker and your microphone, and that path is out of decibri's hands. A path that is stable, direct, and free of clipping gives good reduction. Ordinary laptop speakers or desktop monitor speakers, a few tens of centimetres from the microphone, in a normal room, give modest reduction. That is the common case, and modest is the expected outcome, not a sign that something is broken.

What helps, in rough order: lower the playback volume, increase the distance between speaker and microphone, avoid driving the speaker into distortion, and keep the physical arrangement fixed while capture runs. Moving the laptop mid-call invalidates the alignment.

Speech detection may still trigger on far-end audio

The detector reads after cancellation, so it sees less of your playback than it otherwise would. It does not follow that it stops firing on your playback altogether.

Whether it fires depends on whether the residual echo sits below the room's noise, not on how much was removed. Loud playback in a quiet room can leave a residue that is quiet in absolute terms and still well clear of the noise floor, and the detector will call that speech.

If you are enabling echo cancellation specifically to stop your own output triggering speech detection, it will help, and it will not reliably solve it. The robust approach is to also track when you are playing and treat detections during playback accordingly. Do not build a barge-in design that depends on the detector alone staying quiet through your own output.

Double talk

Double talk is the moment someone cuts in while your own audio is still playing, which is exactly the moment a barge-in design cares about most. With both ends live at once, the canceller holds its adaptation rather than learning from a signal that is no longer only echo. Two consequences: echo reduction during those stretches is lower than when the far end talks alone, and the near-end talker is very slightly affected. doubleTalk in the metrics tells you when this is happening.

Music and sustained tones may never lock

Alignment is found by looking for the far-end signal inside the capture. That needs something aperiodic to lock onto. Speech has it. Music with a steady beat, a held tone, a ringtone, a hum, a test sine can all fail to produce a usable alignment, and delaySamples can stay null for the whole of a musical passage.

This is a real use case and worth planning for. If your far end is a media stream, expect echo cancellation to work during speech and to be unreliable during music. An alignment found during speech does carry across into the music that follows, so a stream that opens with a spoken introduction fares better than one that opens on a note.

Common mistakes

The reference at the wrong rate

Symptom: everything runs, no error is raised, and the echo is untouched. delaySamples stays null while acquisitionParked climbs and referenceSilence does not.

Fix: set referenceSampleRate / reference_sample_rate to the rate of the audio you are pushing, and push it unconverted.

Playback that never reaches your code, and nobody pushed

Symptom: the echo is untouched, delaySamples stays null, and referenceSilence climbs in step with acquisitionParked.

Fix: push at the point where your code hands audio to the player. If the audio never passes through your code, echo cancellation cannot help.

Enabled with headphones

Symptom: no benefit, and a canceller searching for an echo that does not exist.

Fix: leave the option off. Detect the output route if your application supports both, and only enable it for the loudspeaker case.

Pushing before capture is running

Symptom: pushes vanish. referenceDropped stays at zero, because a discarded push is not a dropped sample.

Fix: check that the metrics accessor returns an object before you rely on pushes. In Node, push from inside the playback loop or the data handler.

Handing over a whole utterance in one push

Symptom: referenceDropped jumps by a large count on a single call, and alignment does not appear.

Fix: the queue spans two seconds at the reference rate. Push in blocks, in step with playback.

Pushing a buffer in the wrong sample format

Symptom: in Python, a TypeError naming the mismatch. In Node, no error at all: the bytes are taken as they are and reinterpreted in the microphone's dtype, so a Float32Array pushed into an int16 capture is read as noise and nothing aligns.

Fix: push in the same dtype the microphone was constructed with.

Expecting stop() to preserve the metrics

Symptom: the accessor returns null after capture stops, and a final reading you meant to log is lost.

Fix: read the metrics while capture is still running.

Per-language detail

The exhaustive signatures, the config and metrics types, the async surface, and the error tables live on the binding reference pages:

The same capability is available in the Rust API, where the option and the reference push sit on the capture builder. See the crate documentation for the Rust signatures.

Related