OpenAI

The OpenAI class can be used to access OpenAI models like GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo.

Azure OpenAI Service

If you'd like to use OpenAI models but are concerned about privacy, you can use the Azure OpenAI service, which is GDPR and HIPAA compliant. After applying for access here, you will typically hear back within only a few days. Once you have access, set up a model in config.json like so:

"models": [{
    "title": "Azure OpenAI",
    "provider": "openai",
    "model": "gpt-4",
    "apiBase": "https://my-azure-openai-instance.openai.azure.com/",
    "engine": "my-azure-openai-deployment",
    "apiVersion": "2023-07-01-preview",
    "apiType": "azure",
    "apiKey": "<MY_API_KEY>"
}]

The easiest way to find this information is from the chat playground in the Azure OpenAI portal. Under the "Chat Session" section, click "View Code" to see each of these parameters.

OpenAI compatible servers / APIs

OpenAI compatible servers

• KoboldCpp
• text-gen-webui
• FastChat
• LocalAI
• llama-cpp-python
• TensorRT-LLM

OpenAI compatible APIs

• Anyscale Endpoints
• Anyscale Private Endpoints

If you are using an OpenAI compatible server / API, you can change the apiBase like this:

~/.continue/config.json

{
  "models": [
    {
      "title": "OpenAI-compatible server / API",
      "provider": "openai",
      "model": "MODEL_NAME",
      "apiKey": "EMPTY",
      "apiBase": "http://localhost:8000/v1"
    }
  ]
}

To force usage of chat/completions instead of completions endpoint you can set

"useLegacyCompletionsEndpoint": false

View the source