Skip to main content

Using OpenAI Compatible Providers With Roo Code

Roo Code supports a wide range of AI model providers that offer APIs compatible with the OpenAI API standard. This means you can use models from providers other than OpenAI, while still using a familiar API interface. This includes providers like:

  • Local models running through tools like Ollama and LM Studio (covered in separate sections).
  • Cloud providers like Perplexity, Together AI, Anyscale, and others.
  • Any other provider offering an OpenAI-compatible API endpoint.

This document focuses on setting up providers other than the official OpenAI API (which has its own dedicated configuration page).

General Configuration​

The key to using an OpenAI-compatible provider is to configure two main settings:

  1. Base URL: This is the API endpoint for the provider. It will not be https://api.openai.com/v1 (that's for the official OpenAI API).
  2. API Key: This is the secret key you obtain from the provider.
  3. Model ID: This is the model name of the specific model.

You'll find these settings in the Roo Code settings panel (click the icon):

  • API Provider: Select "OpenAI Compatible".
  • Base URL: Enter the base URL provided by your chosen provider. This is crucial.
  • API Key: Enter your API key.
  • Model: Choose a model.
  • Model Configuration: This lets you customize advanced configuration for the model
    • Max Output Tokens
    • Context Window
    • Image Support
    • Computer Use
    • Input Price
    • Output Price

Native Tool Calling​

Roo Code uses the native tool-calling protocol exclusively. Models must support native tool calling to work with Roo Code.

At a high level:

  • Tool definitions are sent to the model using the native tools schema.
  • Tool calls stream back as dedicated tool events, including the tool name, arguments, and metadata.
  • Tool arguments are streamed incrementally, which reduces latency between the model deciding to use a tool and Roo Code executing it.

Requirements​

For native tool calling to work, the following must be true:

  1. The provider endpoint fully supports native tool calling (OpenAI or an OpenAI-compatible endpoint that implements the tools API).
  2. The selected model supports native tool calling.
Model Compatibility

Models that do not support native tool calling cannot be used with Roo Code. There is no fallback protocol. If you see tool-related errors, verify that your model supports native function/tool calling and that your provider endpoint implements the tools API correctly.

Example: simple native tool flow​

Here's a simplified example of how a file-reading tool might be exposed:

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "read_file",
        "description": "Read a file from the workspace with line numbers.",
        "parameters": {
          "type": "object",
          "properties": {
            "path": { "type": "string", "description": "Relative file path" },
            "start_line": { "type": "integer", "nullable": true },
            "end_line": { "type": "integer", "nullable": true }
          },
          "required": ["path"]
        }
      }
    }
  ]
}

When the model decides to use read_file, Roo Code surfaces streamed tool events in the task timeline:

  • A native tool call event with the tool name and arguments as they're being generated
  • The corresponding tool result event showing the file contents and any truncation or line-range information

This gives you lower-latency feedback on which tools are being used and with what arguments.

Limitations​

  • Model support: Not all models support native tool calling. If a model doesn't support tools, it cannot be used with Roo Code. Check your provider's documentation to confirm tool-calling support for your chosen model.
  • Provider quirks: Some OpenAI-compatible providers only partially implement the native tools API. If you encounter tool-related errors, verify that your provider fully supports the native tools specification.

For a deeper overview of how tools work in Roo Code in general, see the Tool Use Overview.

Troubleshooting​

  • "Invalid API Key": Double-check that you've entered the API key correctly.
  • "Model Not Found": Make sure you're using a valid model ID for your chosen provider.
  • Connection Errors: Verify the Base URL is correct and that your provider's API is accessible.
  • Tool-calling errors: Roo Code requires native tool calling. If your model does not support native tool calling, you must switch to a model that does. There is no XML or fallback tool protocol.
  • Unexpected Results: If you're getting unexpected results, try a different model.

By using an OpenAI-compatible provider, you can leverage the flexibility of Roo Code with a wider range of AI models. Remember to always consult your provider's documentation for the most accurate and up-to-date information.