How to Prepare a Dataset for Whisper Small Fine-Tuning

Preparing a reliable fine-tuning dataset starts with understanding where the base model needs improvement. When we evaluated Whisper Small on technical audio, it struggled with AI model names, technical terms, acronyms, and sentences that combined everyday language with technical vocabulary. The WER results confirmed that these errors followed clear patterns.

We then looked for public datasets containing the language our users typically use, but none provided enough relevant technical vocabulary. Instead, we used hundreds of real conversations between users and our AI agent. These recordings reflected the vocabulary and speaking patterns the model would encounter in the product.

This article explains how we turned 1,000 real call recordings into a clean, human-reviewed dataset, from filtering and segmentation to transcription and quality checks. 

Why We Needed More Than a Quick Fix

Before building the full dataset, we ran a smaller experiment using 100 short audio clips, with 10 training samples for each of the failure cases identified during our evaluation of Whisper Small. The goal was to target the terms the base model struggled with and measure whether fine-tuning improved its transcription accuracy.

The results showed a small improvement: repeated words became less frequent, and the transcripts read more smoothly. However, most of these gains disappeared when we tested the model on unseen audio, suggesting that it had overfit the limited training samples.

This experiment showed us that fine-tuning was promising, but we needed a larger and more varied dataset for the model to generalise effectively. 

What is overfitting? 

Overfitting occurs when a model performs well on its training samples but cannot generalise to unseen data. Our experiment showed that adding more synthetic clips was unlikely to solve this problem. We needed a larger and more varied dataset, so we turned to real call recordings from our product.

What We Were Trying to Achieve?

Our goal was straightforward. We wanted a model that could reliably transcribe real users speaking naturally while interacting with a Voice AI agent built with Whisper and LiveKit. We were not looking for a model that performed well only on a general benchmark, but one that worked reliably with our product and data.

That may seem like a small difference, but it makes a big difference in real life.  A model trained on general speech data gets a little better at everything. A model trained on your own data gets meaningfully better at the one thing that matters to you.

Our goals for this dataset

• Capture the real vocabulary and speaking style from our own product's conversations
• Focus entirely on user voice; the AI agent's voice does not need to be transcribed
• Make sure every transcript was checked and corrected by a real person before being used for training
• Keep things practical; we chose Whisper Small because it is lightweight enough to run efficiently in our setup.

Starting with 1,000 Real Conversations

We started with 1,000 call recordings from our product. Each one was a full conversation between a user and our AI agent, typically anywhere from 15 to 30 minutes long. That is a lot of audio, and in its raw form, it was not usable for training at all.

STT training works best with short, focused audio clips of around 20 to 30 seconds each. So the first challenge was turning those long recordings into something the model could actually learn from. Here is the step-by-step process we followed.

Why Automated Transcripts Still Need Human Review

Automatically generated transcripts provide a useful starting point, but they should be treated as first drafts rather than training-ready data. Human review is still necessary to identify and correct missing, inaccurate, or repeated words before the transcripts are used for fine-tuning.

How We Cleaned the Dataset for Fine-Tuning

Although we had 1,000 transcribed clips, not all of them were suitable for training. Each call contained both the user’s voice and the AI agent’s voice. Since the model needed to transcribe natural user speech, we excluded agent audio and other clips that did not meet our training criteria.

How We Rebalanced the Dataset with User-Voice Clips

After removing more than 500 agent-voice clips, we returned to the original pool of 17,994 clips and selected additional user-voice samples. This restored the dataset size while keeping it aligned with the intended use case: transcribing natural user speech.

Removing agent audio was an important cleaning decision because it prevented synthesised speech from influencing a model intended to recognise human voices. Replacing those clips with relevant user speech improved the dataset’s consistency without lowering its quality.

How We Manually Reviewed Every Audio Sample

This was the slowest part of the whole process, and honestly, the most important one. We listened to every single clip by hand, read through its transcript, and corrected any mistakes we found.

It might be tempting to skip this step or speed through it. Please do not. A model learns from exactly what you put in front of it. If the training transcripts have mistakes in them, the model learns to make those same mistakes. Taking the time to get the transcripts right here pays off directly in how well the model performs later. 

Our Manual Transcript Review Process

What We Learned from Reviewing More Than 1,000 Clips

Spending time with over 1,000 clips manually taught us things that no automated tool would have caught. Here are the patterns we noticed most often during the evaluation process.

Manual review helped us decide what to keep, remove, and correct. These decisions turned the automatically transcribed clips into a dataset suitable for fine-tuning.

Where We Ended Up, The Final Dataset

After all the filtering, chunking, selecting, transcribing, cleaning, and manual reviewing, we had a dataset we felt confident training on. Here is a summary of what we ended up with.

What comes next

With 788 clean, human-reviewed audio clips and 4 hours of real user speech, the dataset was ready for Whisper Small fine-tuning. We then tested different checkpoints and training epochs to understand which configuration produced the best results before running the model in a real-time streaming environment.

A final thought on dataset preparation

Building this dataset took more time than the fine-tuning itself. That is completely normal, and it is actually a sign you are doing things right. The quality of your training data is the biggest factor in whether fine-tuning works. Every hour you spend getting it right here shows up in the model's performance later.

Conclusion

Preparing a training dataset is not the glamorous part of machine learning. But it is the part that everything else depends on. Here are the things we would want anyone reading this to take away. 

•  If you have your own real-world data, use it; it will always be more valuable than a synthetic dataset for your specific use case
• Remove anything that does not match what you actually want to transcribe; in our case, that meant removing all agent voice audio
• Auto-generated transcripts are a useful starting point, but they are not good enough to train on without human review
• Very short clips are usually not worth keeping; they rarely contain enough content to be useful for training
• Manual evaluation is slow, but it is not optional. Every mistake you leave in becomes a mistake the model learns
• Keep your working file updated as you go; you will thank yourself when you need to pick up where you left off.