Notes - Sully.ai

Creating Notes with Sully.ai

The Sully.ai API allows you to turn audio conversations into clinical notes, saving you time and improving your documentation quality. Notes are generated using the API in 3 steps:

Transcribe the conversation audio by uploading a file or streaming audio in real-time.
Generate a note using the transcription.
Retrieve the completed note (via webhooks or polling).

1. Transcribe Audio

Upload a File

You can transcribe audio files using the Create Audio Transcription API. First, submit your file for transcription:

// Transcribe an audio file
const body = new FormData();
body.append('audio', audioBuffer, {
  filename: 'audio.mp3',
  contentType: 'audio/mpeg',
});

const response = await fetch('https://api.sully.ai/v1/audio/transcriptions', {
  method: 'POST',
  headers: { 
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id'
  },
  body
});

const { data: { transcriptionId } } = await response.json();

Then wait for completion either via webhooks or polling:

// Poll until transcription is complete
let response;

do {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  response = await fetch(`https://api.sully.ai/v1/audio/transcriptions/${transcriptionId}`, {
    headers: { 
      'X-Api-Key': 'your-api-key',
      'X-Account-Id': 'your-account-id'
    }
  });
} while (response.data.status === 'STATUS_PROCESSING');

if (response.data.status === 'STATUS_ERROR') {
  throw new Error('Transcription failed');
}

const transcription = response.data.payload?.transcription;

Supported file types include MP3, WAV, M4A, and OGG (max 50MB).

Stream in Real-time

If you are looking to stream audio for transcription during visits, you will first need to request a token from the API, then open a WebSocket so that you can pass audio and receive transcriptions in real-time.

// Get streaming token
const tokenResponse = await fetch('https://api.sully.ai/v1/audio/transcriptions/stream/token', {
  method: 'POST',
  headers: {
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id'
  }
});
const { data: { token } } = await tokenResponse.json();

// Connect to WebSocket
const ws = new WebSocket(
  `wss://api.sully.ai/audio/transcriptions/stream?sample_rate=16000&account_id=${accountId}&api_token=${token}`
);

// Handle connection events
ws.onopen = () => console.log("Connected");
ws.onerror = (error) => console.error("WebSocket error:", error);

// Send audio data
ws.send(JSON.stringify({ audio: base64AudioData }));

// Receive transcription
const transcriptionSegments = [];
let segmentIndex = 0;

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.text) {
    transcriptionSegments[segmentIndex] = data.text;

    // isFinal indicates transcription for the current segment is complete
    if (data.isFinal) {
      segmentIndex++;
    }
  }
};

2. Generate A Note

SOAP Notes

You can generate SOAP notes immediately using Sully’s pre-configured note type.

// Create a SOAP note from a transcription
const response = await fetch('https://api.sully.ai/v1/notes', {
  method: 'POST',
  headers: {
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    transcript: "I've been feeling really tired lately. How long has this been going on? About two weeks. Let's run some blood tests to check your iron and thyroid levels.",
    noteType: { type: 'soap' }
  })
});

const { data: { noteId } } = await response.json();

Custom Note Styles

If Sully’s SOAP notes aren’t what you’re looking for, you can create your own Note Style by providing a sample note for Sully to use as a reference.

// 1. Create a custom note style
const styleResponse = await fetch('https://api.sully.ai/v1/note-styles', {
  method: 'POST',
  headers: {
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    sampleNote: "CC: Headache\nHPI: Patient reports headache for 3 days...",
    instructions: ["Use bullet points for symptoms", "Include vital signs"]
  })
});
const { data: { template } } = await styleResponse.json();

// 2. Use the style for creating notes
const noteResponse = await fetch('https://api.sully.ai/v1/notes', {
  method: 'POST',
  headers: {
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    transcript: 'Patient reports headache for 3 days...',
    noteType: { type: 'note_style', template }
  })
});
const { data: { noteId } } = await noteResponse.json();

Structured Note Templates

If you need a precise note structure that Note Styles aren’t providing, you can instead specify a Note Template. This will allow you to control the fine details of your note formatting such as lists, headers, and section lengths.

// Create a note using a template
const response = await fetch('https://api.sully.ai/v1/notes', {
  method: 'POST',
  headers: {
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    transcript: "I've been feeling really tired lately. How long has this been going on? About two weeks. Let's run some blood tests to check your iron and thyroid levels.",
    noteType: { 
      type: 'note_template'
      template: {
        "id": "my-template",
        "title": "My Template",
        "globalPrompt": "Be concise",
        "sections": [
          // see the Note Template schema for more details
        ]
      }
    }
  })
});

const { data: { noteId } } = await response.json();

3. Retrieve The Note

Notes typically only take a few seconds to generate. You can retrieve your completed note either by polling the notes API or setting up a webhook so that your application may be notified whenever a note or transcription has been completed.

Poll for Notes

// Poll until note is complete
let response;

do {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  response = await fetch(`https://api.sully.ai/v1/notes/${noteId}`, {
    headers: { 
      'X-Api-Key': 'your-api-key',
      'X-Account-Id': 'your-account-id'
    }
  });
} while (response.data.status === 'STATUS_PROCESSING');

if (response.data.status === 'STATUS_ERROR') {
  throw new Error('Note generation failed');
}

const note = response.data.payload;

Webhooks

For full webhook setup instructions, follow the Webhooks guide. Once set up, your webhook handler will receive events related to audio transcription and note generation.

app.post('/webhook', (req, res) => {
  // ... webhook signature handling

  const event = JSON.parse(req.body);

  switch(event.type) {
    // Receive a successfully generated note
    case 'note_generation.succeeded':
      const { id: noteId, payload } = event.data;
      break;
    // Handle a note generation error
    case 'note_generation.failed':
      const { id: noteId, error } = event.data;
      break;
    // Receive a successfully generated transcription
    case 'audio_transcription.succeeded':
      const { transcriptionId, payload: { transcription } } = event.data;
      break;
    // Handle an audio transcription error
    case 'audio_transcription.failed':
      const { transcriptionId, error } = event.data;
      break;
  }

  res.status(200).send('Webhook received');
});

Language Support

You can specify one of our supported BCP47 language tags both when uploading and streaming audio for transcription. Audio which is not in the specified language will be ignored. Or, you can also specify the multi tag as the language for multilingual conversations.

// Transcribe an audio file
const body = new FormData();
body.append('audio', audioBuffer, {
  filename: 'audio.mp3',
  contentType: 'audio/mpeg',
});

// Transcribe a conversation in US Spanish
const response = await fetch('https://api.sully.ai/v1/audio/transcriptions?language=es-US', {
  method: 'POST',
  headers: { 
    'X-Api-Key': 'your-api-key',
    'X-Account-Id': 'your-account-id'
  },
  body
});

const { data: { transcriptionId } } = await response.json();

Specify More Context

If you have additional context programmatically available, you can provide it to Sully’s note creation API so that it can be incorporated into your notes.

date: The date of the patient visit.
patientInfo: High-level patient descriptors including name, date of birth, and gender.
medicationList: A list of known patient medications.
previousNote: A previous note needed as context for this note (e.g. if it is a follow-up).
instructions: Additional instructions for how you’d like the note to be generated.
context: Any additional context you’d like to provide.

See the Create Note API for the up-to-date list of options.

Guides

​Creating Notes with Sully.ai

​1. Transcribe Audio

​Upload a File

​Stream in Real-time

​2. Generate A Note

​SOAP Notes

​Custom Note Styles

​Structured Note Templates

​3. Retrieve The Note

​Poll for Notes

​Webhooks

​Language Support

​Specify More Context