Input

Configure audio formats and language hints for a Speechmatics Realtime transcription session.

Supported input audio formats

Sessions can be configured to use two types of audio input: file and raw.
We recommend using the raw option, unless you have a specific reason to use the file option.

For capturing raw audio in the browser, try our browser-audio-input package.

audio_format

The format must be supplied in the audio_format field of the StartRecognition message. See the API reference.

AudioFormatIn the API reference: StartRecognition

AudioFormatRaw | AudioFormatFile

one of

AudioFormatRaw

Raw

optional

Raw audio samples, described by the following additional mandatory fields:

type

"raw"

encoding

RawAudioEncodingEnum"pcm_f32le""pcm_s16le""mulaw"

sample_rate

integer

The sample rate of the audio in Hz.

AudioFormatFile

File

optional

Choose this option to send audio encoded in a recognized format. The AddAudio messages have to provide all the file contents, including any headers. The file is usually not accepted all at once, but segmented into reasonably sized messages.

Note: Only the following formats are supported: wav, mp3, aac, ogg, mpeg, amr, m4a, mp4, flac

type

"file"

Language hints

Language hints are optional and apply only to the Melia 1 and Oak 1 models, which detect supported languages automatically. For the Enhanced and Standard models, set language to a supported language instead.

Set language_hints in the StartRecognition message to the supported languages you expect in the audio. Hints have two effects:

• Language labels are restricted to the hinted languages. Every word in the transcript is labeled with one of the languages you list. With one hint, every word gets that label. With two or more, labels can switch between the hinted languages.
• Transcription is biased toward the hinted languages. This can improve accuracy, especially for short utterances and similar-sounding languages. Speech in other languages is less likely to be transcribed, but can still appear in the transcript, labeled as one of the hinted languages.

This config hints that the audio contains English and Arabic:

{
  "message": "StartRecognition",
  "audio_format": {
    "type": "raw",
    "encoding": "pcm_s16le",
    "sample_rate": 16000
  },
  "transcription_config": {
    "model": "melia-1",
    // Melia 1 requires "multi" rather than a single language code
    "language": "multi",
    "language_hints": ["en", "ar"]
  }
}

Hints restrict language labels rather than only biasing detection: the model cannot label a word with a language you did not hint. If your audio may contain languages you cannot predict, omit language_hints.

To render each detected language correctly, refer to the LanguageInfo message.

Sending audio

After receiving a RecognitionStarted message, you can start sending audio over the Websocket connection. Audio is sent as binary data, encoded in the format specified in the StartRecognition message. See Protocol overview for complete details of the API protocol.

Next steps

View our guides:

• using a microphone to learn how to capture audio from a microphone.
• using FFMPEG to find out how to pipe microphone audio to the API.