Real-Time Call Center Transcription: Over 90% Cost Reduction with Better Features
After six months with Azure OpenAI’s Whisper API, a major call center needed a better solution. They were processing hundreds of thousands of minutes of audio per day, and costs were climbing. More importantly, they needed speaker diarization—something the OpenAI API simply didn’t offer.
Without diarization, they couldn’t answer critical performance management questions: What percentage of agents proactively offered a specific solution? What percentage mentioned key product features? Which agents needed coaching on particular topics? These insights were locked behind the technical limitation of not knowing who said what.
This is the story of how they moved to Carolina Cloud and cut their transcription costs by over 90% in absolute terms while gaining the critical features they couldn’t get before.
The Setup: A Good Start with Growing Pains
Our customer came to us in an enviable position. Their call center infrastructure was already well-organized: audio files in .wav format were automatically dumped to a shared drive in a clean file structure. The raw materials were there and ready to process.
They also had a substantial enterprise agreement with Azure, which gave them immediate access to OpenAI’s Whisper API through the Microsoft-OpenAI partnership. For their MVP phase, this was perfect—clean, simple, and at $0.006 per minute of audio, seemingly affordable.
But as their call volume grew, the cracks began to show.
The Breaking Point: When MVP Meets Reality
The problems started stacking up:
Rate limits hit at exactly the wrong time. The team tried Python’s threading to fire multiple requests simultaneously, but the bottleneck was on Azure’s backend—no amount of client-side parallelization could help. They couldn’t simply subscribe directly to OpenAI due to onerous legal and compliance requirements that would take months to navigate. Carolina Cloud eliminated this concern entirely—we operate in a SOC2 and HIPAA-compliant Tier 3 data center in North Carolina (huge shoutout to Tier.Net for helping us achieve this).
Transcription slowed during peak demand. When call volume spiked, processing times dragged. The API’s orchestration overhead meant significant delays before audio files even reached a Microsoft GPU.
The cost curve was unsustainable. Here’s where the math got scary: in their industry, call volume could increase tenfold overnight based on interest rate fluctuations. With per-minute pricing, costs would 10x exactly when they needed transcription the most.
No speaker diarization. This was the real dealbreaker. For performance management and quality assurance, they needed to analyze agent behavior: What percentage of agents mentioned the return policy? What percentage proactively offered the premium service? Which agents needed coaching on objection handling? Without knowing who was saying what in each call, these critical metrics were impossible to calculate. The OpenAI API couldn’t do it, and their locked-in Azure relationship made it difficult to adopt third-party diarization services quickly.
Death by a thousand ingress fees. Microsoft charges for data ingress from on-premises to Azure, and these fees added up daily—seemingly trivial until you’re transcribing thousands of calls per day.
The team had Linux and Python expertise, but they’d never worked with NVIDIA MIG (Multi-Instance GPU) technology. They needed a partner who could help them level up.
Enter Carolina Cloud: Testing the Waters
When the customer approached us, we knew the solution would involve NVIDIA GPUs and local processing. But first, we needed to prove the economics worked.
Using nvtop to monitor GPU utilization, we noticed that each transcription job used less than 1/7th of the full H100’s capacity. This confirmed that MIG partitioning was the right approach—by splitting the GPU into seven instances, we could genuinely process seven calls in parallel without any instance becoming a bottleneck. The result: true 7x performance improvement.
We set them up with a single NVIDIA H100 server configured with MIG to create seven isolated GPU instances—seven parallel “listeners” that could process calls simultaneously. Each listener ran in its own Docker container with dedicated access to 1/7th of the H100’s GPU resources. Every container ran Whisper locally with NVIDIA NeMo for speaker diarization.
The results were immediate and dramatic:
- Processing speed increased 7x compared to API calls during peak times—each MIG instance handling transcription in parallel
- Zero rate limiting—all seven instances could run at full capacity simultaneously
- Built-in diarization with NeMo’s state-of-the-art models
- Predictable, fixed costs regardless of volume spikes
- Zero ingress fees—no charges for transferring data from on-premises
The Numbers Tell the Story
Here’s how the two solutions compare on the metrics that matter:
| Metric | Azure OpenAI Whisper API | Carolina Cloud | Advantage |
|---|---|---|---|
| Cost Model | $0.006 per minute (variable) | Fixed monthly cost | Over 90% absolute cost reduction |
| Cost Scaling at Peak (10x volume) | Scales linearly (10x cost) | Same fixed cost | Cost stays flat |
| Effective Cost per Minute | $0.006 | ~48x lower | 48x cheaper |
| Speaker Diarization | ❌ Not available | ✅ NVIDIA NeMo | Enables performance metrics |
| Processing Architecture | Sequential API calls (single queue) | 7 parallel Docker containers via MIG | 7x throughput increase |
| Rate Limiting | ⚠️ Unknown limits that were hit often | ✅ Far higher and known limit | Predictable performance |
| Data Transfer Fees | Microsoft ingress charges (on-prem → Azure) | No ingress fees | Additional savings |
| Technical Support | Standard Azure support portal | Direct access to engineers who configured MIG, optimized Docker containers, and trained their team | Partner vs. vendor |
Making the Switch
The transition took less than a week. We helped them:
- Set up the H100 server with MIG configured for optimal transcription workloads
- Deploy Docker containers for each MIG instance running Whisper with NVIDIA NeMo diarization
- Integrate with their existing file structure and workflow
- Train their team on managing the MIG instances and Docker containers
The team went from zero MIG experience to confidently managing their own GPU infrastructure. Today, they process hundreds of thousands of call minutes daily, with speaker attribution, at a fraction of their previous cost.
The Bottom Line
Sometimes the cloud isn’t the answer. Sometimes you need dedicated hardware, expert support, and a partner who understands your specific needs.
An over 90% absolute cost reduction (48x cheaper per minute) isn’t just a number—at this scale, the savings add up quickly. But for this call center, moving from Azure OpenAI’s API to Carolina Cloud meant more than just cost savings:
- Essential features unlocked: Speaker diarization with NeMo enabled the performance metrics they desperately needed
- True 7x performance improvement: Seven parallel MIG instances processing calls simultaneously with dedicated GPU resources
- Predictable, fixed costs: No more worrying about 10x cost spikes during volume surges
- Freedom from constraints: No rate limits, no vendor lock-in, all processing on their infrastructure
- Room to scale: Current setup handles peak volumes comfortably with capacity to spare
Want to see if Carolina Cloud can deliver similar results for your workload? Get in touch and let’s run the numbers together.
A Note on This Case Study
We want to be transparent: the Whisper API has evolved significantly since this migration, with improved rate limits and performance. There are also now several excellent third-party services offering diarization capabilities.
This was one of our more hands-on projects—it required a few smart engineers on our client’s side who were willing to roll up their sleeves and work closely with us. We also offer simple on-demand GPU rentals for teams that just need infrastructure without the white-glove service. Different needs, different solutions.
But for this client, the hands-on approach was exactly what made this a win-win. Their company had been looking to build more in-house technical capabilities anyway, and this project gave their team the opportunity to gain hands-on experience with GPU infrastructure, MIG partitioning, and containerized ML workloads. They weren’t just cutting costs; they were investing in their team’s growth.
This kind of personal investment in customer success—where we’re not just providing infrastructure but actively partnering to upskill their team—is what sets Carolina Cloud apart. We’re not for everyone, but for teams that want to own their infrastructure and build deep technical capabilities, we’re the right partner.