Streaming chat completions

Set streamMode to true and pass an onChunk callback to client.streamChat().

Stream a completion


import { ServiceProvider } from '@preternatural-js/client-sdk'
 
const result = await client.streamChat(
  {
    messages: [
      {
        role: 'user',
        content: 'Write a short description of Saturn.'
      }
    ],
    scope: {
      provider: ServiceProvider.openai,
      modelIdentifier: 'gpt-4o-mini',
      productId: 'your-product-id'
    },
    temperature: 0.7,
    maxTokens: 500,
    streamMode: true
  },
  chunk => {
    appendToInterface(chunk)
  }
)

Read the final text

The current return type is string | ChatCompletion. Narrow it before using the value as text:


const result = await client.streamChat(request, chunk => {
  appendToInterface(chunk)
})
 
if (typeof result !== 'string') {
  throw new Error('Expected streamed text')
}
 
console.log('Complete response:', result)

Cancel the request

Pass an AbortSignal on the request:


import {
  CustomerStreamCancelledError,
  ServiceProvider
} from '@preternatural-js/client-sdk'
 
const controller = new AbortController()
 
stopButton.addEventListener(
  'click',
  () => controller.abort(),
  { once: true }
)
 
try {
  await client.streamChat(
    {
      messages: [
        {
          role: 'user',
          content: 'Write a long essay about Saturn.'
        }
      ],
      scope: {
        provider: ServiceProvider.openai,
        modelIdentifier: 'gpt-4o-mini',
        productId: 'your-product-id'
      },
      temperature: 0.7,
      maxTokens: 2_000,
      streamMode: true,
      signal: controller.signal
    },
    chunk => {
      appendToInterface(chunk)
    }
  )
} catch (error) {
  if (!(error instanceof CustomerStreamCancelledError)) {
    throw error
  }
}

A caller-initiated cancellation rejects with CustomerStreamCancelledError.

Return a `ReadableStream`

Server-side code can use streamServerSideChat() to obtain a web-standard ReadableStream<string>:


const stream = client.streamServerSideChat({
  messages: [
    {
      role: 'user',
      content: 'Write a short description of Saturn.'
    }
  ],
  scope: {
    provider: ServiceProvider.openai,
    modelIdentifier: 'gpt-4o-mini',
    productId: 'your-product-id'
  },
  temperature: 0.7,
  maxTokens: 500,
  streamMode: true
})

Read it with a stream reader:


const reader = stream.getReader()
 
try {
  while (true) {
    const { done, value } = await reader.read()
 
    if (done) {
      break
    }
 
    appendToInterface(value)
  }
} finally {
  reader.releaseLock()
}

Canceling the returned stream aborts the gateway request:


await reader.cancel()

streamServerSideChat() always sends a streaming provider request. Its streamMode input does not change that behavior.

Handle customer access changes

The SDK watches gateway errors while a stream is active.

If the customer becomes revoked, suspended, or deleted, the SDK clears the customer session and terminates any ongoing streams:


import {
  CustomerAccessInvalidatedError
} from '@preternatural-js/client-sdk'
 
try {
  await client.streamChat(request, handleChunk)
} catch (error) {
  if (error instanceof CustomerAccessInvalidatedError) {
    console.error(error.reason)
    redirectToSignIn()
  } else {
    throw error
  }
}