Skip to main content

LocalAI Embeddings

LocalAI Setup​

LocalAI is a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format.

To use LocalAI Embeddings within CiniterFlow, follow the steps below:

  1. git clone https://github.com/go-skynet/LocalAI
  2. cd LocalAI
  3. LocalAI provides an API endpoint to download/install the model. In this example, we are going to use BERT Embeddings model:

  1. In the /models folder, you should be able to see the downloaded model in there:

  1. You can now test the embeddings:
curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{
    "input": "Test",
    "model": "text-embedding-ada-002"
  }'
  1. Response should looks like:

CiniterFlow Setup​

Drag and drop a new LocalAIEmbeddings component to canvas:

Fill in the fields:

  • Base Path: The base url from LocalAI such as http://localhost:8080/v1
  • Model Name: The model you want to use. Note that it must be inside /models folder of LocalAI directory. For instance: text-embedding-ada-002

That's it! For more information, refer to LocalAI docs.