Streaming chat completions

Pass an onText callback to client.chat.stream() to receive each piece of text as it arrives. Use client.chat.streamText() or client.chat.streamChunks() when you need a ReadableStream.

Stream a completion


import { ServiceProvider } from '@preternatural-js/client-sdk'
 
const result = await client.chat.stream(
  {
    messages: [
      {
        role: 'user',
        content: 'Write a short description of Saturn.'
      }
    ],
    scope: {
      provider: ServiceProvider.openai,
      modelIdentifier: 'gpt-4o-mini',
      productId: 'your-product-id'
    },
    temperature: 0.7,
    maxTokens: 500
  },
  chunk => {
    appendToInterface(chunk)
  }
)

Read the final text

The promise resolves to the whole text when the stream ends:


const result = await client.chat.stream(request, chunk => {
  appendToInterface(chunk)
})
 
console.log('Complete response:', result)

Cancel the request

Pass an AbortSignal on the request:


import {
  CustomerStreamCancelledError,
  ServiceProvider
} from '@preternatural-js/client-sdk'
 
const controller = new AbortController()
 
stopButton.addEventListener(
  'click',
  () => controller.abort(),
  { once: true }
)
 
try {
  await client.chat.stream(
    {
      messages: [
        {
          role: 'user',
          content: 'Write a long essay about Saturn.'
        }
      ],
      scope: {
        provider: ServiceProvider.openai,
        modelIdentifier: 'gpt-4o-mini',
        productId: 'your-product-id'
      },
      temperature: 0.7,
      maxTokens: 2_000,
      signal: controller.signal
    },
    chunk => {
      appendToInterface(chunk)
    }
  )
} catch (error) {
  if (!(error instanceof CustomerStreamCancelledError)) {
    throw error
  }
}

When your own signal cancels the stream, the promise rejects with CustomerStreamCancelledError.

Return a `ReadableStream`

client.chat.streamText() returns a web-standard ReadableStream<string> of the text only:


const stream = client.chat.streamText({
  messages: [
    {
      role: 'user',
      content: 'Write a short description of Saturn.'
    }
  ],
  scope: {
    provider: ServiceProvider.openai,
    modelIdentifier: 'gpt-4o-mini',
    productId: 'your-product-id'
  },
  temperature: 0.7,
  maxTokens: 500
})

Read it with a stream reader:


const reader = stream.getReader()
 
try {
  while (true) {
    const { done, value } = await reader.read()
 
    if (done) {
      break
    }
 
    appendToInterface(value)
  }
} finally {
  reader.releaseLock()
}

Cancelling the returned stream cancels the gateway request. Call cancel() on the reader before you release its lock:


await reader.cancel()

client.chat.streamText() always sends a streaming provider request.

Read the raw chunks

client.chat.streamChunks() returns a ReadableStream of OpenAI ChatCompletionChunk objects. Each chunk keeps its tool calls and other fields, including usage, which the gateway always requests on a streaming completion.


const chunks = client.chat.streamChunks(request)
const reader = chunks.getReader()
 
try {
  while (true) {
    const { done, value } = await reader.read()
 
    if (done) {
      break
    }
 
    handleChunk(value)
  }
} finally {
  reader.releaseLock()
}

client.chat.stream() and client.chat.streamText() read this stream and keep only the text.

Handle customer access changes

If the customer is revoked, deleted, or suspended, the SDK rejects the stream with CustomerAccessInvalidatedError, clears the session, and ends the session’s other streams. Replacing or clearing the session ends its streams with CustomerSessionChangedError. Other errors reject the stream unchanged. See Gateway errors.


import {
  CustomerAccessInvalidatedError
} from '@preternatural-js/client-sdk'
 
try {
  await client.chat.stream(request, handleChunk)
} catch (error) {
  if (error instanceof CustomerAccessInvalidatedError) {
    console.error(error.reason)
    redirectToSignIn()
  } else {
    throw error
  }
}