Streaming

If streaming is set when making prediction, tokens will be sent as data-only server-sent events as they become available.

Using Python/TS Library

CiniterFlow provides 2 libraries:

• Python: pip install ciniterflow
• Typescript: npm install ciniterflow-sdk

Python

from ciniterflow import CiniterFlow, PredictionData

def test_streaming():
    client = CiniterFlow()

    # Test streaming prediction
    completion = client.create_prediction(
        PredictionData(
            chatflowId="<flow-id>",
            question="Tell me a joke!",
            streaming=True
        )
    )

    # Process and print each streamed chunk
    print("Streaming response:")
    for chunk in completion:
        # {event: "token", data: "hello"}
        print(chunk)


if __name__ == "__main__":
    test_streaming()

Typescript

import { CiniterFlowClient } from 'ciniterflow-sdk'

async function test_streaming() {
  const client = new CiniterFlowClient({ baseUrl: 'http://localhost:3000' });

  try {
    // For streaming prediction
    const prediction = await client.createPrediction({
      chatflowId: '<flow-id>',
      question: 'What is the capital of France?',
      streaming: true,
    });

    for await (const chunk of prediction) {
        // {event: "token", data: "hello"}
        console.log(chunk);
    }
    
  } catch (error) {
    console.error('Error:', error);
  }
}

// Run streaming test
test_streaming()

cURL

curl https://localhost:3000/api/v1/predictions/{flow-id} \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Hello world!",
    "streaming": true
  }'

event: token
data: Once upon a time...

A prediction's event stream consists of the following event types:

Event	Description
start	The start of streaming
token	Emitted when the prediction is streaming new token output
error	Emitted when the prediction returns an error
end	Emitted when the prediction finishes
metadata	All metadata such as chatId, messageId, of the related flow. Emitted after all tokens have finished streaming, and before end event
sourceDocuments	Emitted when the flow returns sources from vector store
usedTools	Emitted when the flow used tools

Streamlit App

https://github.com/HenryHengZJ/ciniterflow-streamlit