Streaming

This API provides streaming speech-to-text transcriptions using WebSockets.

Endpoint:
wss://api.sully.ai/v1/audio/transcriptions/stream?account_id=1234567890&api_token=1234567890&sample_rate=16000&language=en

Headers

X-API-KEY · string

The API key to use for authentication. Required if X-API-TOKEN is not provided.

X-API-TOKEN · string

The API token to use for authentication. Required if X-API-KEY is not provided.

X-ACCOUNT-ID · string

The account ID to use for authentication.

Query parameters

language · string · default: "en"

The language of your submitted audio. See our Supported Languages documentation for a complete list of language options.

sample_rate · string

The sample rate of your submitted audio.

encoding · string

Specifies the encoding format of the audio being sent.

Important: This parameter is required when transmitting raw, unstructured audio packets without headers. If the audio data is encapsulated within a container format, this parameter should be omitted.

Supported formats:

• linear16: 16-bit, little-endian PCM audio
• flac: Free Lossless Audio Codec (FLAC)
• mulaw: Mu-law encoded WAV
• amr-nb: Adaptive Multi-Rate, narrowband
• amr-wb: Adaptive Multi-Rate, wideband
• opus: Ogg Opus codec
• speex: Speex codec
• g729: G729 codec (usable with raw or containerized audio)

account_id · string

The account ID to use for authentication. Required if X-ACCOUNT-ID is not provided.

api_token · string

A temporary authentication token. Required if X-API-KEY is not provided.

When to use

The Speech-to-Text Websockets API is designed to generate text from partial audio input. It’s well-suited for scenarios where the input audio is being streamed or generated in chunks.

Protocol

The WebSocket API uses a bidirectional protocol that encodes all messages as JSON objects.

Connection Status Messages

Upon successful connection, the server sends a status message:

{
  "status": "connected",
}

When the connection closes:

{
  "status": "disconnected",
}

Important: Wait for the “status”: “connected” message before sending audio data. This ensures the server is ready to process your stream.

Streaming input audio

The client can send messages with audio input to the server. The messages can contain the following fields:

{
  "audio": "Y3VyaW91cyBtaW5kcyB0aGluayBhbGlrZSA6KQ=="
}

audio · string · required

A generated partial audio chunk encoded as a base64 string.

Browser MediaRecorder Notice: When using Chrome’s MediaRecorder API, the first audio chunk contains critical header information. Always send this first chunk for proper audio processing. Failing to include header information may result in transcription errors or complete failure.

Streaming output audio

The server will always respond with a message containing the following fields:

Response message

{
  "type": "transcript",
  "audio_start": 0,
  "audio_end": 3.2,
  "duration": 3.2,
  "text": "Hello, world",
  "isFinal": false, 
  "is_final": false,
  "words": [
    {
      "word": "Hello",
      "start": 0.2,
      "end": 0.7,
      "confidence": 0.9856743,
      "punctuated_word": "Hello,"
    },
    {
      "word": "world",
      "start": 0.9,
      "end": 1.2,
      "confidence": 0.9978341,
      "punctuated_word": "world"
    }
  ],
  "timestamp": "2023-08-15T14:32:07.123Z"
}

type · string

The type of response, will be “transcript” for transcription results.

audio_start · number

Start time of the audio segment in seconds.

audio_end · number

End time of the audio segment in seconds.

duration · number

Duration of the audio segment in seconds.

text · string

The processed text sequence.

isFinal · boolean · deprecated

Indicates if the generation is complete. Deprecated: Use is_final instead.

is_final · boolean

Indicates if the generation is complete.

words · array

Array of word objects with text content and timing information:

• word: The raw word as recognized
• start: Start time of the word in seconds
• end: End time of the word in seconds
• confidence: Confidence score between 0-1 for the word recognition
• punctuated_word: The word with proper capitalization and punctuation

timestamp · string

ISO-formatted timestamp when the response was generated.