Docs
File Transcription

File Transcription

Implementing file-based speech-to-text in your applications

File transcription processes complete audio files offline, unlike real-time transcription that processes audio online in a streaming fashion.

  1. Audio Capture: Recording or selecting a complete audio file
  2. Full-Context Processing: Analyzing the entire audio file at once
  3. One-Time Output: Producing a complete transcription after processing finishes

Basic Example

Pro SDK

Argmax Pro SDK includes the WhisperKitPro framework that implements file transcription:

import Argmax
 
// Initialize Argmax SDK to enable Pro access
await ArgmaxSDK.with(ArgmaxConfig(apiKey: "ax_*****"))
 
let config = WhisperKitProConfig(model: "large-v3-v20240930")
let whisperKitPro = try await WhisperKitPro(config)
let transcript = try? await whisperKitPro.transcribe(audioPath: "path/to/audio.m4a").text

Open-source SDK

Argmax Open-source SDK includes the WhisperKit framework that implements file transcription:

import WhisperKit
 
let config = WhisperKitConfig(model: "large-v3-v20240930")
let whisperKit = try await WhisperKit(config)
let transcript = try? await whisperKit.transcribe(audioPath: "path/to/audio.m4a").text

Advanced Examples

Record-then-transcribe

/// Record audio to a temporary file for batch processing
func startFileRecording() {
    guard !isProcessing && !isStreaming else { return }
 
    if audioProcessor == nil {
        audioProcessor = AudioProcessor()
    }
 
    guard let audioProcessor = self.audioProcessor else {
        processingError = "AudioProcessor is not initialized"
        return
    }
 
    stopRequestedAt = nil
 
    Task {
        await MainActor.run {
            isProcessing = true
            isStreaming = true
            processingError = nil
            transcriptionResult = ""
            hypothesisText = ""
            startAudioLevelTimer()
        }
 
        do {
            try audioProcessor.startRecordingLive()
        } catch {
            await MainActor.run {
                isProcessing = false
                isStreaming = false
                processingError = "Failed to start recording: \(error.localizedDescription)"
                stopAudioLevelTimer()
            }
        }
    }
}
 
/// Stops recording to file and transcribes the recorded audio
@discardableResult
func stopFileRecording(useProVersion: Bool) async -> URL? {
    guard isStreaming, let audioProcessor = self.audioProcessor else { return nil }
 
    await MainActor.run {
        isStreaming = false
        stopAudioLevelTimer()
    }
 
    let audioSamples = Array(audioProcessor.audioSamples)
    audioProcessor.stopRecording()
 
    let tempURL = FileManager.default.temporaryDirectory.appendingPathComponent("recording-\(Date().timeIntervalSince1970).wav")
 
    let success = saveAudioSamplesToFile(audioSamples, url: tempURL)
 
    if success {
        await transcribeAudio(url: tempURL, useProVersion: useProVersion, useStreaming: false)
        return tempURL
    } else {
        await MainActor.run {
            processingError = "Failed to save audio file"
            isProcessing = false
        }
        return nil
    }
}
 
/// Saves audio samples to a WAV file
private func saveAudioSamplesToFile(_ samples: [Float], url: URL) -> Bool {
    let sampleRate = 16000
    let channelCount = 1
 
    guard let format = AVAudioFormat(
        commonFormat: .pcmFormatFloat32,
        sampleRate: Double(sampleRate),
        channels: AVAudioChannelCount(channelCount),
        interleaved: false
    ) else {
        print("Failed to create audio format")
        return false
    }
 
    guard let buffer = AVAudioPCMBuffer(
        pcmFormat: format,
        frameCapacity: AVAudioFrameCount(samples.count)
    ) else {
        print("Failed to create audio buffer")
        return false
    }
 
    for i in 0..<min(samples.count, Int(buffer.frameCapacity)) {
        buffer.floatChannelData?[0][i] = samples[i]
    }
    buffer.frameLength = AVAudioFrameCount(min(samples.count, Int(buffer.frameCapacity)))
 
    do {
        let audioFile = try AVAudioFile(
            forWriting: url,
            settings: format.settings,
            commonFormat: .pcmFormatFloat32,
            interleaved: false
        )
 
        try audioFile.write(from: buffer)
        return true
    } catch {
        print("Failed to save audio file: \(error)")
        return false
    }
}

Advanced Features

Pro Models

Pro SDK offers significantly faster and more energy-efficient models. These models also lead to higher accuracy word-level timestamps.

To upgrade, simply apply this diff to your initial configuration code:

- let config = WhisperKitConfig(model: "large-v3-v20240930")
+ let config = WhisperKitProConfig(
+     model: "large-v3-v20240930",
+     modelRepo: "argmaxinc/whisperkit-pro",
+     modelToken: "hf_*****" // Request access at https://huggingface.co/argmaxinc/whisperkit-pro
+ )

For now, you need to request model access here. We are working on removing this extra credential requirement.

VAD-based Audio Chunking

Audio files that are longer than 30 seconds are processed in chunks. Naive chunking with 30 second intervals (.chunkingStrategy = none) may lead to middle-of-speech cuts and (ii) extended silence in the beginning of an audio chunk, both of which are known to lead to lower quality transcriptions.

Voice Activity Detection (VAD) is built into the Pro SDK to help find precise seek points for chunking to improve the transcription accuracy even further. This feature downloads the 1.5 MB SpeakerSegmenter model which accurately separates voice from non-voice audio segments at ~16 ms resolution. Set .chunkingStrategy = .modelVAD to activate this Pro SDK feature.

Open-source SDK implements the same VAD feature based on an "audio energy"-based function that does not rely on a deep learning model. You may set .chunkingStrategy = .vad to activate this Open-source SDK feature.

TODO(rik,andrey)

Multi-Channel Audio

Both WhisperKit and WhisperKitPro support multi-channel audio processing, which can be useful when working with audio files containing multiple speakers or audio sources.

The SDK allows you to specify how to handle multi-channel audio:

  1. Default Behavior: Merges all channels into a mono track for processing
  2. Channel Selection: Allows selecting specific channels for transcription
  3. Channel Summing: Combines selected channels with normalization

Here's how to configure WhisperKit to use specific audio channels:

let config = WhisperKitConfig(
    // Other configuration options...
    audioInputConfig: AudioInputConfig(channelMode: .sumChannels([1, 3, 5]))
)

Multi-channel audio processing is particularly useful for:

  1. Interview Recordings: Where different microphones may capture different speakers
  2. Meeting Transcription: Where table microphones may be positioned to capture different participants
  3. Simplified Speaker Separation: If your audio file has distinct speakers in different channels (e.g., one speaker per channel)
  4. Audio Quality Enhancement: Selecting channels with the clearest audio and discarding noisy channels

The audio merging algorithm works as follows:

  1. Finds the peak amplitude across all channels
  2. Checks if the peak of the mono (summed) version is higher than any of the peaks of the individual channels
  3. Normalizes the combined track so that the peak of the mono channel matches the peak of the loudest channel

This approach ensures the merged audio maintains appropriate volume levels while combining information from multiple channels.

UI Considerations

When implementing file transcription in your application, consider these UI elements:

Recording Indicator

Visual feedback when recording is active

TODO(rik): Either code snippet or screenshot from our example

Progress Indicator

TODO(rik): Either code snippet or screenshot from our example

Results Preview

TODO(rik): Either code snippet or screenshot from our example

Editor

Allow users to review and correct transcription results

Downgrading to Open-SourceReal-time Transcription