AI SDK OpenAI-Compatible Providers: Setup, Tools and Memory

To use an OpenAI-compatible endpoint with the AI SDK, install @ai-sdk/openai-compatible, call createOpenAICompatible with a name, an API key and a baseURL that ends in the version prefix (usually /v1), then pick a model from that instance. "Compatible" only means the endpoint accepts the same request shape. It does not promise that streaming, tool calls or structured outputs work the same way. This guide covers the setup, a checklist for verifying an endpoint, how to define tools with schemas the model can rely on, and what you have to build yourself for memory.

Nexusbloom

Keep AI Tool Schemas Consistent

Keep AI tool schemas consistent across web, CLI, and API surfaces with one manifest and one pure function.

Explore Nexusbloom

TL;DR

Use createOpenAICompatible with a baseURL that includes /v1. A missing prefix is the most common cause of a 404 that looks like a dead endpoint.
Compatible does not mean identical. Check text generation, streaming, tool calling while streaming, structured outputs, reasoning content and system messages one by one.
A tool is a description, an input schema and an execute function. Keep the schema in one place so every surface that calls the tool validates the same way.
MCP tools can be loaded into the SDK, but their schemas are only as good as the server that publishes them.
Memory is not built in. Your application must store, retrieve and trim history on every request.

What "OpenAI-compatible" means in the AI SDK
Connect a compatible endpoint step by step
What to verify before you call an endpoint compatible
Define tools with schemas the model can trust
MCP tools and AI SDK tools
Memory: what the SDK does not do for you
Common errors on compatible endpoints
Our perspective
Try the one-file approach
A practical workflow
FAQ
Sources

What "OpenAI-compatible" means in the AI SDK

An OpenAI-compatible API accepts the same HTTP requests as OpenAI's chat endpoint and returns the same JSON shape. Many hosting services, local model servers and internal gateways copy that contract so that existing clients keep working. The AI SDK has a provider package for this case, @ai-sdk/openai-compatible, which lets you point the SDK at any such endpoint without a dedicated provider.

The difference between this package and the first-party OpenAI provider matters. The first-party provider knows OpenAI's exact features. The compatible provider assumes only the common denominator and lets you switch extras on yourself. That is why two endpoints that both say "compatible" can behave differently under the same code.

Connect a compatible endpoint step by step

The package page shows the pattern. This is the same shape with a placeholder host:

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';

const provider = createOpenAICompatible({
  name: 'example',
  baseURL: 'https://api.example.com/v1',
  apiKey: process.env.MY_API_KEY,
});

const { text } = await generateText({
  model: provider.chatModel('your-model-id'),
  prompt: 'Summarize this changelog in three bullets.',
});

Work through it in this order:

Install both packages. You need ai and @ai-sdk/openai-compatible.
Set the base URL with the version prefix. Use https://host/v1, not https://host. The SDK appends the route, so a bare domain produces a 404 that is easy to misread as a network or DNS problem.
Keep the key server-side. Read it from an environment variable on the server. Never ship it to the browser.
Choose the model method. chatModel targets the chat completions route. The package also exposes methods for completion, embedding and image models, but whether a given host serves them is up to the host.
Run one plain generateText call first. Add streaming, tools and structured output one at a time afterward, so you know which feature broke.

If you maintain several environments, validate the variables before the process starts. A free .env validator catches a missing key or malformed URL before it becomes a confusing runtime error.

Headers and custom query parameters

Some gateways need an extra header or a query parameter such as an API version. The provider settings accept custom headers, so pass them at creation time rather than patching requests later. If options seem to be silently ignored, test the same request with a plain HTTP client. That tells you whether the SDK dropped the option or the server did.

What to verify before you call an endpoint compatible

The SDK's own provider documentation lists what the compatible provider supports: text generation, streaming, tool calling with streaming, structured outputs when enabled, reasoning content, system messages and provider-dependent multimodal input. Whether your specific host honors each item is a separate question. Vercel's gateway documentation makes the same distinction: capability support, request fields and response metadata vary by API, and a model supporting a capability is not the same as a surface exposing it.

Use this checklist and run each row as its own test.

Capability	What to test	Typical symptom when it fails
Plain text	One `generateText` call	404, 401, or a model-not-found error
Streaming	`streamText` and read the full stream	Output arrives all at once, or the stream ends early
System messages	A `system` prompt that changes the answer	The instruction is ignored or merged oddly
Tool calling	A tool the model must call	The model answers in prose and never calls the tool
Tools while streaming	The same tool through `streamText`	Arguments arrive broken or incomplete
Structured output	A schema-bound response with `supportsStructuredOutputs` enabled	Output is valid JSON but ignores the schema
Reasoning content	A reasoning model, if you use one	Reasoning text leaks into the answer or disappears
Multimodal input	An image part, only if you need it	Request rejected or part silently dropped

The structured-output row deserves care. The setting supportsStructuredOutputs tells the SDK the endpoint can enforce a schema. Turning it on for a host that cannot do so does not create the capability. It just moves the failure from setup to runtime.

A small test harness

Write these checks once as a script and run it against every new endpoint and every time you change models. Each case should assert on the structure of the result (a tool was called, the JSON parsed, the stream produced more than one chunk) rather than on exact wording. Model text changes between runs, but structure should not.

Three structural checks for endpoint tests

Define tools with schemas the model can trust

Tools are the next thing most people search for after setup. In the AI SDK a tool is an object with a few parts. The DeepWiki overview of the repository describes them as a description, an input schema (Zod or JSON Schema) that both the model and the SDK use to validate arguments, an optional context schema, an execute handler that receives validated inputs, and an optional mapper that turns the result into text or binary content for the model.

That structure explains most tool bugs:

The description is a prompt. The model decides whether to call the tool from it. Say what the tool does and when to use it.
The schema is the contract. Name fields plainly, mark required ones, add short descriptions to each property, and use enums for closed sets. Vague schemas produce vague arguments.
execute should be a pure function where possible. Take validated input, return a result, touch nothing else. It is then easy to test without a model in the loop.

Tools whose types are unknown at compile time

Sometimes you cannot know the schema when you write the code. The SDK's dynamicTool function exists for that case. Its reference page lists MCP tools without schemas, user-defined functions loaded at runtime, tools from external sources or databases, and tools generated from user input. Dynamic tools trade compile-time safety for flexibility, so validate what they return before you pass it onward.

Schema drift is the real reliability problem

The hardest tool bug is not a typo. It is the same logic implemented in three places. The agent tool takes userId, the REST endpoint takes user_id, and the CLI flag is --user. One gets a new required field and the others do not. Weeks later the agent behaves differently from the web page, and nobody can say which one is right.

Two habits prevent this. Keep one JSON Schema for the input and derive every surface's validation from it. Keep the logic in one function that every surface calls. You can build that yourself, and many teams do. Our approach is described in the next sections.

Quick tip: when you hand-write schemas, a JSON formatter and a JSON creator make it quicker to inspect and draft the shape before it goes into a tool definition.

MCP tools and AI SDK tools

The Model Context Protocol (MCP) gives agents a standard way to discover and call tools hosted elsewhere. The AI SDK has a client for it in @ai-sdk/mcp. Vercel's page on using MCP with the AI SDK shows the flow: create an MCP client with an HTTP transport and optional auth headers, fetch its tools, and pass them to generateText alongside your model. Run that code on the server or in a terminal so credentials stay private.

When you integrate an MCP server, check these things:

Does each tool publish an input schema? Tools without schemas fall into the dynamic case above, and the model gets less guidance on arguments.
Are the descriptions written for a model? A terse human-facing label gives the model little to decide on.
How are results shaped? Large payloads cost tokens. Map them to the smallest useful text.
How is auth handled? Pass credentials through transport headers or an auth provider, never in the prompt.
Does the tool behave the same as your other surfaces? If your web page and your MCP tool run different code, they will diverge.

That last check is where MCP integrations quietly fail. Publishing a tool to an MCP directory does not keep it consistent with your API.

Memory: what the SDK does not do for you

The AI SDK sends the messages you give it. It does not decide what to remember between requests. For an agent to retain context, your application has to do four jobs on every turn:

Load the relevant history or notes for this user or session from your own storage.
Send them as part of the messages, within the model's context window.
Save the new user message and the model's reply, including tool calls and tool results, after the turn.
Trim or summarize old turns so the history does not grow without bound.

Two patterns cover most apps. Short-term memory is the recent message list for one conversation. Long-term memory is a store of facts about a user that you retrieve selectively and insert into the system prompt. Keep them separate, because they age differently and need different deletion rules.

Two cautions apply when you test against a compatible endpoint. First, replay tool calls and results in the same order and format you received them, because some hosts reject a history where a tool result has no matching call. Second, store user data under your privacy rules, with a way to delete it on request.

Common errors on compatible endpoints

Most problems fall into a handful of patterns.

404 on every call. Check the base URL for the missing /v1 first.
401 or 403. The key is for a different host or project, or the host expects a custom header.
Tools never called. The model may not support tool calling, or the host drops the tools field. Test with the raw HTTP request.
Stream stops mid-answer. Some hosts end streams differently when a tool call is in flight. Test streaming with and without tools.
Structured output ignores the schema. Confirm the host enforces schemas, not just JSON mode.
Memory looks broken. Print the exact messages array you send. Usually history was never loaded or was truncated.

Debug by removing variables. One model, one prompt, no tools, no streaming. Add back one feature at a time.

Our perspective

Every compatibility problem above comes from the same root: the same behavior defined in several places. An agent tool, an API route and a CLI command each have their own schema, and they drift.

We built NexusBloom around the opposite idea. You write one file, a JSON Schema manifest plus one pure function, and we serve it as a web page, a CLI command, an API endpoint and an AI agent tool, with the same schema and the same execution on each. The model sees the schema you wrote, and your web users and scripts hit the same logic.

This matters most for people shipping agent integrations and MCP tools. When the schema is a single source of truth, you stop debugging why a tool behaves one way in your editor and another way in production. You spend your time on the logic instead of the glue.

Benjamin

Try the one-file approach

If you maintain the same logic across a page, a CLI, an API and an agent tool, NexusBloom lets you define it once and serve it everywhere. You can start on the Free plan, and the Creator plan is 9 EUR per month and the Pro plan is 29 EUR per month. See the pricing page for what each plan includes.

Nexusbloom

If you want a feel for what a single-source tool looks like before you commit, browse the free online tools, or request a new tool you would like to see.

A practical workflow

Putting it together, this is the sequence we suggest for a new agent integration:

Write the input schema and a pure function first, with tests that need no model.
Connect your provider and run the capability table above.
Wrap the function as an AI SDK tool using that same schema.
Expose the same function through your API and CLI from the same schema.
Add memory last, so you can tell whether failures come from history or from tools.
Re-run your checks whenever you change the model or host.

FAQ

Do I need a dedicated provider package for every OpenAI-compatible service?

No. The compatible provider package covers any service that follows the chat completions contract. You only need to supply the name, base URL and key. A dedicated package is useful when a service has features beyond the common contract.

Why do I get a 404 when my key is correct?

The most common cause is a base URL without the version prefix. Use the full path such as https://api.example.com/v1. If that is already right, check that the model ID exists on that host and that the host serves the route you are calling.

Does the AI SDK remember earlier messages automatically?

No. It sends the messages you pass in each call. Your application must load stored history, send it, save the new turn and trim older content. Decide what is short-term conversation and what is long-term user data, and store each accordingly.

Are MCP tools the same as AI SDK tools?

They are related but not identical. The SDK's MCP client loads tools from a server and exposes them for use in calls. Those tools may arrive without typed schemas, so validate inputs and outputs, and confirm they behave like your other surfaces.