API Reference - OpenAI API

Audio

Learn how to turn audio into text or text into audio.

Related guide: Speech to text

Create speech

post https://api.openai.com/v1/audio/speech

Generates audio from the input text.

Request body

input

string

Required

The text to generate audio for. The maximum length is 4096 characters.

model

string

Required

One of the available TTS models: tts-1, tts-1-hd or gpt-4o-mini-tts.

voice

string

Required

The voice to use when generating the audio. Supported voices are alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, and verse. Previews of the voices are available in the Text to speech guide.

instructions

string

Optional

Control the voice of your generated audio with additional instructions. Does not work with tts-1 or tts-1-hd.

response_format

string

Optional

Defaults to mp3

The format to audio in. Supported formats are mp3, opus, aac, flac, wav, and pcm.

speed

number

Optional

Defaults to 1

The speed of the generated audio. Select a value from 0.25 to 4.0. 1.0 is the default.

stream_format

string

Optional

Defaults to audio

The format to stream the audio in. Supported formats are sse and audio. sse is not supported for tts-1 or tts-1-hd.

Returns

The audio file content or a stream of audio events.

Example request

curl

1
2
3
4
5
6
7
8
9
curl https://api.openai.com/v1/audio/speech \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini-tts",
    "input": "The quick brown fox jumped over the lazy dog.",
    "voice": "alloy"
  }' \
  --output speech.mp3

1
2
3
4
5
6
7
8
9
10
from pathlib import Path
import openai

speech_file_path = Path(__file__).parent / "speech.mp3"
with openai.audio.speech.with_streaming_response.create(
  model="gpt-4o-mini-tts",
  voice="alloy",
  input="The quick brown fox jumped over the lazy dog."
) as response:
  response.stream_to_file(speech_file_path)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import fs from "fs";
import path from "path";
import OpenAI from "openai";

const openai = new OpenAI();

const speechFile = path.resolve("./speech.mp3");

async function main() {
  const mp3 = await openai.audio.speech.create({
    model: "gpt-4o-mini-tts",
    voice: "alloy",
    input: "Today is a wonderful day to build something people love!",
  });
  console.log(speechFile);
  const buffer = Buffer.from(await mp3.arrayBuffer());
  await fs.promises.writeFile(speechFile, buffer);
}
main();

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
using System;
using System.IO;

using OpenAI.Audio;

AudioClient client = new(
    model: "gpt-4o-mini-tts",
    apiKey: Environment.GetEnvironmentVariable("OPENAI_API_KEY")
);

BinaryData speech = client.GenerateSpeech(
    text: "The quick brown fox jumped over the lazy dog.",
    voice: GeneratedSpeechVoice.Alloy
);

using FileStream stream = File.OpenWrite("speech.mp3");
speech.ToStream().CopyTo(stream);

Create transcription

post https://api.openai.com/v1/audio/transcriptions

Transcribes audio into the input language.

Request body

file

Required

The audio file object (not file name) to transcribe, in one of these formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm.

model

string

Required

ID of the model to use. The options are gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1 (which is powered by our open source Whisper V2 model), and gpt-4o-transcribe-diarize.

chunking_strategy

"auto" or object

Optional

Controls how the audio is cut into chunks. When set to "auto", the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. server_vad object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. Required when using gpt-4o-transcribe-diarize for inputs longer than 30 seconds.

include

array

Optional

Additional information to include in the transcription response. logprobs will return the log probabilities of the tokens in the response to understand the model's confidence in the transcription. logprobs only works with response_format set to json and only with the models gpt-4o-transcribe and gpt-4o-mini-transcribe. This field is not supported when using gpt-4o-transcribe-diarize.

known_speaker_names

array

Optional

Optional list of speaker names that correspond to the audio samples provided in known_speaker_references[]. Each entry should be a short identifier (for example customer or agent). Up to 4 speakers are supported.

known_speaker_references

array

Optional

Optional list of audio samples (as data URLs) that contain known speaker references matching known_speaker_names[]. Each sample must be between 2 and 10 seconds, and can use any of the same input audio formats supported by file.

language

string

Optional

The language of the input audio. Supplying the input language in ISO-639-1 (e.g. en) format will improve accuracy and latency.

prompt

string

Optional

An optional text to guide the model's style or continue a previous audio segment. The prompt should match the audio language. This field is not supported when using gpt-4o-transcribe-diarize.

response_format

string

Optional

Defaults to json

The format of the output, in one of these options: json, text, srt, verbose_json, vtt, or diarized_json. For gpt-4o-transcribe and gpt-4o-mini-transcribe, the only supported format is json. For gpt-4o-transcribe-diarize, the supported formats are json, text, and diarized_json, with diarized_json required to receive speaker annotations.

stream

boolean

Optional

Defaults to false

If set to true, the model response data will be streamed to the client as it is generated using server-sent events. See the Streaming section of the Speech-to-Text guide for more information.

Note: Streaming is not supported for the whisper-1 model and will be ignored.

temperature

number

Optional

Defaults to 0

The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit.

timestamp_granularities

array

Optional

Defaults to segment

The timestamp granularities to populate for this transcription. response_format must be set verbose_json to use timestamp granularities. Either or both of these options are supported: word, or segment. Note: There is no additional latency for segment timestamps, but generating word timestamps incurs additional latency. This option is not available for gpt-4o-transcribe-diarize.

Returns

The transcription object, a diarized transcription object, a verbose transcription object, or a stream of transcript events.

Example request

curl

1
2
3
4
5
curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F file="@/path/to/file/audio.mp3" \
  -F model="gpt-4o-transcribe"

1
2
3
4
5
6
7
8
from openai import OpenAI
client = OpenAI()

audio_file = open("speech.mp3", "rb")
transcript = client.audio.transcriptions.create(
  model="gpt-4o-transcribe",
  file=audio_file
)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import fs from "fs";
import OpenAI from "openai";

const openai = new OpenAI();

async function main() {
  const transcription = await openai.audio.transcriptions.create({
    file: fs.createReadStream("audio.mp3"),
    model: "gpt-4o-transcribe",
  });

  console.log(transcription.text);
}
main();

1
2
3
4
5
6
7
8
9
10
11
12
13
using System;

using OpenAI.Audio;
string audioFilePath = "audio.mp3";

AudioClient client = new(
    model: "gpt-4o-transcribe",
    apiKey: Environment.GetEnvironmentVariable("OPENAI_API_KEY")
);

AudioTranscription transcription = client.TranscribeAudio(audioFilePath);

Console.WriteLine($"{transcription.Text}");

Response

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that.",
  "usage": {
    "type": "tokens",
    "input_tokens": 14,
    "input_token_details": {
      "text_tokens": 0,
      "audio_tokens": 14
    },
    "output_tokens": 45,
    "total_tokens": 59
  }
}

Create translation

post https://api.openai.com/v1/audio/translations

Translates audio into English.

Request body

file

Required

The audio file object (not file name) translate, in one of these formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm.

model

string or "whisper-1"

Required

ID of the model to use. Only whisper-1 (which is powered by our open source Whisper V2 model) is currently available.

prompt

string

Optional

An optional text to guide the model's style or continue a previous audio segment. The prompt should be in English.

response_format

string

Optional

Defaults to json

The format of the output, in one of these options: json, text, srt, verbose_json, or vtt.

temperature

number

Optional

Defaults to 0

Returns

The translated text.

Example request

curl

1
2
3
4
5
curl https://api.openai.com/v1/audio/translations \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F file="@/path/to/file/german.m4a" \
  -F model="whisper-1"

1
2
3
4
5
6
7
8
from openai import OpenAI
client = OpenAI()

audio_file = open("speech.mp3", "rb")
transcript = client.audio.translations.create(
  model="whisper-1",
  file=audio_file
)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import fs from "fs";
import OpenAI from "openai";

const openai = new OpenAI();

async function main() {
    const translation = await openai.audio.translations.create({
        file: fs.createReadStream("speech.mp3"),
        model: "whisper-1",
    });

    console.log(translation.text);
}
main();

1
2
3
4
5
6
7
8
9
10
11
12
13
14
using System;

using OpenAI.Audio;

string audioFilePath = "audio.mp3";

AudioClient client = new(
    model: "whisper-1",
    apiKey: Environment.GetEnvironmentVariable("OPENAI_API_KEY")
);

AudioTranscription transcription = client.TranscribeAudio(audioFilePath);

Console.WriteLine($"{transcription.Text}");

Response

1
2
3
{
  "text": "Hello, my name is Wolfgang and I come from Germany. Where are you heading today?"
}

The transcription object (JSON)

Represents a transcription response returned by model, based on the provided input.

logprobs

array

The log probabilities of the tokens in the transcription. Only returned with the models gpt-4o-transcribe and gpt-4o-mini-transcribe if logprobs is added to the include array.

text

string

The transcribed text.

usage

object

Token usage statistics for the request.

OBJECT The transcription object (JSON)

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that.",
  "usage": {
    "type": "tokens",
    "input_tokens": 14,
    "input_token_details": {
      "text_tokens": 10,
      "audio_tokens": 4
    },
    "output_tokens": 101,
    "total_tokens": 115
  }
}

The transcription object (Diarized JSON)

Represents a diarized transcription response returned by the model, including the combined transcript and speaker-segment annotations.

duration

number

Duration of the input audio in seconds.

segments

array

Segments of the transcript annotated with timestamps and speaker labels.

task

string

The type of task that was run. Always transcribe.

text

string

The concatenated transcript text for the entire audio input.

usage

object

Token or duration usage statistics for the request.

OBJECT The transcription object (Diarized JSON)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "task": "transcribe",
  "duration": 42.7,
  "text": "Agent: Thanks for calling OpenAI support.\nCustomer: Hi, I need help with diarization.",
  "segments": [
    {
      "type": "transcript.text.segment",
      "id": "seg_001",
      "start": 0.0,
      "end": 5.2,
      "text": "Thanks for calling OpenAI support.",
      "speaker": "agent"
    },
    {
      "type": "transcript.text.segment",
      "id": "seg_002",
      "start": 5.2,
      "end": 12.8,
      "text": "Hi, I need help with diarization.",
      "speaker": "A"
    }
  ],
  "usage": {
    "type": "duration",
    "seconds": 43
  }
}

The transcription object (Verbose JSON)

Represents a verbose json transcription response returned by model, based on the provided input.

duration

number

The duration of the input audio.

language

string

The language of the input audio.

segments

array

Segments of the transcribed text and their corresponding details.

text

string

The transcribed text.

usage

object

Usage statistics for models billed by audio input duration.

words

array

Extracted words and their corresponding timestamps.

OBJECT The transcription object (Verbose JSON)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "task": "transcribe",
  "language": "english",
  "duration": 8.470000267028809,
  "text": "The beach was a popular spot on a hot summer day. People were swimming in the ocean, building sandcastles, and playing beach volleyball.",
  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 0.0,
      "end": 3.319999933242798,
      "text": " The beach was a popular spot on a hot summer day.",
      "tokens": [
        50364, 440, 7534, 390, 257, 3743, 4008, 322, 257, 2368, 4266, 786, 13, 50530
      ],
      "temperature": 0.0,
      "avg_logprob": -0.2860786020755768,
      "compression_ratio": 1.2363636493682861,
      "no_speech_prob": 0.00985979475080967
    },
    ...
  ],
  "usage": {
    "type": "duration",
    "seconds": 9
  }
}

Stream Event (speech.audio.delta)

Emitted for each chunk of audio data generated during speech synthesis.

audio

string

A chunk of Base64-encoded audio data.

type

string

The type of the event. Always speech.audio.delta.

OBJECT Stream Event (speech.audio.delta)

1
2
3
4
{
  "type": "speech.audio.delta",
  "audio": "base64-encoded-audio-data"
}

Stream Event (speech.audio.done)

Emitted when the speech synthesis is complete and all audio has been streamed.

type

string

The type of the event. Always speech.audio.done.

usage

object

Token usage statistics for the request.

OBJECT Stream Event (speech.audio.done)

1
2
3
4
5
6
7
8
{
  "type": "speech.audio.done",
  "usage": {
    "input_tokens": 14,
    "output_tokens": 101,
    "total_tokens": 115
  }
}

Stream Event (transcript.text.delta)

Emitted when there is an additional text delta. This is also the first event emitted when the transcription starts. Only emitted when you create a transcription with the Stream parameter set to true.

delta

string

The text delta that was additionally transcribed.

logprobs

array

The log probabilities of the delta. Only included if you create a transcription with the include[] parameter set to logprobs.

segment_id

string

Identifier of the diarized segment that this delta belongs to. Only present when using gpt-4o-transcribe-diarize.

type

string

The type of the event. Always transcript.text.delta.

OBJECT Stream Event (transcript.text.delta)

1
2
3
4
{
  "type": "transcript.text.delta",
  "delta": " wonderful"
}

Stream Event (transcript.text.segment)

Emitted when a diarized transcription returns a completed segment with speaker information. Only emitted when you create a transcription with stream set to true and response_format set to diarized_json.

end

number

End timestamp of the segment in seconds.

string

Unique identifier for the segment.

speaker

string

Speaker label for this segment.

start

number

Start timestamp of the segment in seconds.

text

string

Transcript text for this segment.

type

string

The type of the event. Always transcript.text.segment.

OBJECT Stream Event (transcript.text.segment)

1
2
3
4
5
6
7
8
{
  "type": "transcript.text.segment",
  "id": "seg_002",
  "start": 5.2,
  "end": 12.8,
  "text": "Hi, I need help with diarization.",
  "speaker": "A"
}

Stream Event (transcript.text.done)

Emitted when the transcription is complete. Contains the complete transcription text. Only emitted when you create a transcription with the Stream parameter set to true.

logprobs

array

The log probabilities of the individual tokens in the transcription. Only included if you create a transcription with the include[] parameter set to logprobs.

text

string

The text that was transcribed.

type

string

The type of the event. Always transcript.text.done.

usage

object

Usage statistics for models billed by token usage.

OBJECT Stream Event (transcript.text.done)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "type": "transcript.text.done",
  "text": "I see skies of blue and clouds of white, the bright blessed days, the dark sacred nights, and I think to myself, what a wonderful world.",
  "usage": {
    "type": "tokens",
    "input_tokens": 14,
    "input_token_details": {
      "text_tokens": 10,
      "audio_tokens": 4
    },
    "output_tokens": 31,
    "total_tokens": 45
  }
}