Preparing a reliable fine-tuning dataset starts with understanding where the base model needs improvement. When we evaluated Whisper Small on technical audio, it struggled with AI model names, technical terms, acronyms, and sentences that combined everyday language with technical vocabulary. The WER results confirmed that these errors followed clear patterns.
We then looked for public datasets containing the language our users typically use, but none provided enough relevant technical vocabulary. Instead, we used hundreds of real conversations between users and our AI agent. These recordings reflected the vocabulary and speaking patterns the model would encounter in the product.
This article explains how we turned 1,000 real call recordings into a clean, human-reviewed dataset, from filtering and segmentation to transcription and quality checks.
Why We Needed More Than a Quick Fix
Before building the full dataset, we ran a smaller experiment using 100 short audio clips, 10 training samples for each of the 10 failure cases identified during evaluation. The goal was to target the terms Whisper Small struggled with and measure whether fine-tuning improved its transcription accuracy.
The results showed a small improvement: repeated words became less frequent, and the transcripts read more smoothly. However, most of these gains disappeared when we tested the model on unseen audio, suggesting that it had overfit the limited training samples.
This experiment showed us that fine-tuning was promising, but we needed a larger and more varied dataset for the model to generalise effectively.
What is overfitting?
Overfitting occurs when a model performs well on its training samples but cannot generalise to unseen data. Our experiment showed that adding more synthetic clips was unlikely to solve this problem. We needed a larger and more varied dataset, so we turned to real call recordings from our product.
What We Were Trying to Achieve?
Our goal was straightforward. We wanted a model that could reliably transcribe the conversations happening in our product, with real users speaking naturally, in the context of interacting with an AI agent. Not a model that did well on a general benchmark, but one that worked well for us with our data.
That may seem like a small difference, but it makes a big difference in real life. A model trained on general speech data gets a little better at everything. A model trained on your own data gets meaningfully better at the one thing that matters to you.
Our goals for this dataset
- Capture the real vocabulary and speaking style from our own product's conversations
- Focus entirely on user voice; the AI agent's voice does not need to be transcribed
- Make sure every transcript was checked and corrected by a real person before being used for training
- Keep things practical; we chose Whisper Small because it is lightweight enough to run efficiently in our setup.
Walk away with actionable insights on AI adoption.
Limited seats available!
Starting with 1,000 Real Conversations
We started with 1,000 call recordings from our product. Each one was a full conversation between a user and our AI agent, typically anywhere from 15 to 30 minutes long. That is a lot of audio, and in its raw form, it was not usable for training at all.
STT training works best with short, focused audio clips of around 20 to 30 seconds each. So the first challenge was turning those long recordings into something the model could actually learn from. Here is the step-by-step process we followed.
| 01 | Filter out broken recordingsBefore doing anything else, we went through every file to make sure it actually worked. Any recording that was silent, corrupted, or incomplete was removed straight away. There is no point in spending time on audio that cannot be used. |
02 | Split each recording into short clips We cut every long recording into clips of 20 to 30 seconds. This is the format that Whisper-based models work best with. After this step, we had 17,994 individual audio clips, more than 10 hours of audio in total. |
03 | Select 1,000 clips for closer review From nearly 18,000 clips, we picked 1,000 for the next stage. We tried to choose a good mix of different speakers, topics, and audio quality levels, so the dataset would reflect the real variety of conversations in our product. |
04 | Generate transcripts automatically We used Groq's speech recognition pipeline with Whisper Large v3 to create a first-draft transcript for each of the 1,000 clips. These were saved alongside the audio files in a CSV spreadsheet. This gave us a solid starting point, even though the transcripts still needed review and correction. |
Why Automated Transcripts Still Need Human Review
Automatically generated transcripts provide a useful starting point, but they should be treated as first drafts rather than training-ready data. Human review is still necessary to identify and correct missing, inaccurate, or repeated words before the transcripts are used for fine-tuning.
How We Cleaned the Dataset for Fine-Tuning
Although we had 1,000 transcribed clips, not all of them were suitable for training. Each call contained both the user’s voice and the AI agent’s voice. Since the model needed to transcribe natural user speech, we excluded agent audio and other clips that did not meet our training criteria.
| What we removed | Why did we remove it |
Agent voice audio (500+ clips) | We only want the model to learn from user speech. Agent audio pulls it in the wrong direction. |
Clips shorter than 8 seconds | Very short clips almost never contain a complete thought or sentence. They are not useful for training. |
Non-English audio | This project didn't include any clips that weren't in English because our product is focused on English. |
Corrupted or silent files | These were caught early in the filtering step and removed before any further processing. |
How We Rebalanced the Dataset with User-Voice Clips
After removing more than 500 agent-voice clips, we returned to the original pool of 17,994 clips and selected additional user-voice samples. This restored the dataset size while keeping it aligned with the intended use case: transcribing natural user speech.
Removing agent audio was an important cleaning decision because it prevented synthesised speech from influencing a model intended to recognise human voices. Replacing those clips with relevant user speech improved the dataset’s consistency without lowering its quality.
How We Manually Reviewed Every Audio Sample
This was the slowest part of the whole process, and honestly, the most important one. We listened to every single clip by hand, read through its transcript, and corrected any mistakes we found.
It might be tempting to skip this step or speed through it. Please do not. A model learns from exactly what you put in front of it. If the training transcripts have mistakes in them, the model learns to make those same mistakes. Taking the time to get the transcripts right here pays off directly in how well the model performs later.
Our Manual Transcript Review Process
| 01 | Listen to the whole clipWe listened to each clip from start to finish, no skipping ahead. It only takes a moment to miss something, and a missed error becomes a mistake the model will learn from. |
02 | Read the transcript while listening We read the auto-generated transcript word by word alongside the audio. Any time something did not match, a wrong word, a missing word, or an extra word, we noted it and fixed it. |
03 | Make corrections We added any missing words, replaced anything that was wrong, took out repeated words (a very common issue with auto-transcription), and tidied up the punctuation so it reflected how the person actually spoke. |
04 | Save and move on Each corrected transcript was saved straight back into the CSV. We did not wait until the end to update things; keeping the file current meant we never lost our work and always knew exactly where we were. |
What We Learned from Reviewing More Than 1,000 Clips
Spending time with over 1,000 clips manually taught us things that no automated tool would have caught. Here are the patterns we noticed most often during the evaluation process.
| What we noticed | What it meant for the dataset |
A lot of agent-heavy clips | Without manual review, the agent's voice would have quietly degraded the training data. You cannot catch this with automation alone. |
Transcripts drifting on names | Auto-generated transcripts often got proper nouns and hesitations wrong. These were not random mistakes; they happened consistently. |
Very short clips were rarely useful | Clips under 8 seconds almost never contained a complete thought. Removing them improved the overall quality of the dataset. |
Punctuation was inconsistent | Whisper Large v3 sometimes added punctuation in the wrong place or left it out entirely. Every instance needed a manual check. |
Repeated words at clip boundaries | A common auto-transcription artifact, the last word of one segment or the first word of the next would sometimes appear twice. All of these were corrected. |
Manual review helped us decide what to keep, remove, and correct. These decisions turned the automatically transcribed clips into a dataset suitable for fine-tuning.
Walk away with actionable insights on AI adoption.
Limited seats available!
Where We Ended Up, The Final Dataset
After all the filtering, chunking, selecting, transcribing, cleaning, and manual reviewing, we had a dataset we felt confident training on. Here is a summary of what we ended up with.
| What | How much |
Total audio clips in the final dataset | 788 clips |
Total duration of audio | 4 hours |
Clips reviewed by a human | 1,000+ |
Transcripts manually verified | 100% of the dataset |
Type of voice in the dataset | User voice only , agent audio removed |
Format | CSV with paired audio references and transcripts |
What comes next
With 788 clean, human-reviewed audio clips and 4 hours of real user speech, the dataset was ready to be handed over to the fine-tuning stage. The next blog in this series covers what happened when we actually trained on this data, which checkpoint gave us the best results, what we learned about how many training epochs to use, and what happened when we tried to run the model in a real-time streaming environment.
A final thought on dataset preparation
Building this dataset took more time than the fine-tuning itself. That is completely normal, and it is actually a sign you are doing things right. The quality of your training data is the biggest factor in whether fine-tuning works. Every hour you spend getting it right here shows up in the model's performance later.
Conclusion
Preparing a training dataset is not the glamorous part of machine learning. But it is the part that everything else depends on. Here are the things we would want anyone reading this to take away.
- If you have your own real-world data, use it; it will always be more valuable than a synthetic dataset for your specific use case
- Remove anything that does not match what you actually want to transcribe; in our case, that meant removing all agent voice audio
- Auto-generated transcripts are a useful starting point, but they are not good enough to train on without human review
- Very short clips are usually not worth keeping; they rarely contain enough content to be useful for training
- Manual evaluation is slow, but it is not optional. Every mistake you leave in becomes a mistake the model learns
- Keep your working file updated as you go; you will thank yourself when you need to pick up where you left off.
Walk away with actionable insights on AI adoption.
Limited seats available!
