Teaching a Model to Know When You're Done Speaking

The problem

When you talk to a voice agent, something has to decide when you are done. Answer too early and it interrupts you. Wait too long and the silence gets awkward. That decision is called turn detection, and I wanted to see how far I could get predicting it from audio alone: is the speaker finished (an endpoint), or just pausing?

The data

I worked through about 20k streamed samples. The labels were nicely balanced, roughly half endpoint and half continue. The audio was very multilingual - English, Hindi, Spanish, Russian and more. One thing that shaped everything that followed: the spoken text was missing for every sample I inspected. This was going to be an audio-only problem. No cheating with transcripts.

Some early exploration was already interesting. Utterance duration was nearly identical across the two classes (7.67s vs 7.59s), so "longer means finished" was out. End fillers were strongly associated with continuation, and the energy at the very end of the audio was lower for completed turns. Final pause duration turned out to be a weak signal.

Experiment 1: features I could write down by hand

The obvious first attempt: describe each clip with numbers I understood, then let XGBoost sort it out. I computed 12 features across four groups - energy (mean, std, max, final, slope), pitch (mean, std, slope), silence (mean, max, count) and duration - and trained on an 80/20 stratified split.

Result: 0.690 F1 and 0.753 ROC-AUC. A decent lightweight baseline. But the ablation study is where the real story was.

Ablation study: ROC-AUC by feature group — ROC-AUC with only one feature group at a time

Energy alone scored 0.749, almost the full model's 0.753. Duration alone scored 0.482, which is below random. Silence barely moved. So nearly everything the model knew about turn completion, it had learned from how loud the audio was and how that loudness changed. Pitch added almost nothing on top of energy. Every group was telling the same story: on this dataset, my handcrafted features were carrying roughly the same loudness signal. So instead of writing feature number thirteen, I tried a different representation of the audio.

Experiment 2: borrow ears from Whisper

Instead of describing the audio myself, I let a model that had already listened to a lot of speech describe it for me. I took a pretrained Whisper Tiny encoder, kept it completely frozen, mean-pooled its output over time, and trained XGBoost on top of that embedding. No Whisper parameters were updated.

Pipeline: audio to Whisper Tiny encoder to mean pooling to XGBoost

The jump was immediate. With 2,000 training samples it already beat the handcrafted model: 0.805 ROC-AUC versus 0.753. And it kept improving as I fed it more data.

Performance by training set size: ROC-AUC and F1 at 2k, 5k and 10k samples — Whisper embeddings + XGBoost, by training set size

At 10,000 samples it reached 0.848 ROC-AUC. The pretrained encoder was capturing conversational cues that energy, pitch and silence statistics simply do not express.

A free win: move the threshold

One thing bothered me. From 5,000 to 10,000 samples, ROC-AUC went up (0.842 to 0.848) but F1 at the default 0.50 threshold actually went down (0.777 to 0.761). The model was ranking examples better while the fixed cutoff was working against it. So I tuned the classification threshold on validation predictions. The optimum landed at 0.36, and F1 moved from 0.761 to 0.787. No retraining, just asking "done?" a little more readily.

Where it landed

Final comparison of the three models on F1 and ROC-AUC — Same test split throughout. The last bar is not a new model - it is the 10k Whisper model with its decision threshold moved to 0.36, tuned on validation predictions. The handcrafted baseline was trained on an 80/20 split of the full data, so training sizes differ across bars.

What I'd tell you before you trust these numbers

The 2k/5k/10k points are the same pipeline trained on different-sized slices and measured on the same test split - a scaling curve, not three independent validations.
The 0.787 F1 comes from tuning the decision threshold on validation predictions of the same 10k model. That is standard practice, but it means 0.787 is a tuned number, not a fresh held-out result; a new validation pass could move it.
The F1 dip between 5k and 10k samples is real and I don't fully know its cause. The ranking metric kept improving; the default threshold just stopped being the right one.
Whisper Tiny is frozen here and the classifier is XGBoost. This is a feature-extraction setup, not an end-to-end model. Fine-tuning or a bigger encoder would likely do better.
All numbers are on this dataset's test split: multilingual, audio-only clips. A live, streaming voice agent would need its own evaluation before trusting any of this.

The conclusion

I went in expecting to iterate on features for a while. Instead, the ablation told me within one experiment that energy was carrying nearly everything. In this project, the gains came from switching representation rather than adding more handcrafted features: frozen Whisper embeddings took ROC-AUC from 0.753 to 0.848 on the same test split, and tuning the threshold moved the 10k model's F1 from 0.761 to 0.787.

If you're working on a perception problem and your handcrafted features have plateaued, try borrowing a pretrained model's representations before writing feature number thirteen.

The full notebooks, reports and inference code are in the repo: github.com/prateek-mehra/turn-detection.