go-llms

A powerful and flexible Go library for interacting with Large Language Models (LLMs) with built-in support for function calling and streaming responses. Currently supports Anthropic, Google, OpenAI compatible providers, and TypeSafe System One models.

Features

• Supports Anthropic, Google (Gemini + Vertex), and OpenAI (Chat Completions + Responses + WebSocket)
• Also supports using custom endpoints compatible with any of the APIs above
• TypeSafe System One models (Jev): typed judgments with probabilities, driven by a JSON output schema
• Streaming responses (including thinking) for real-time interaction
• Built-in tool calling with Go generics to generate JSON schemas automatically
• Structured output (JSON schema based) for model response
• OpenAI only: Custom grammars like Lark / Regex for structured output
• Prompt cache hints
• Image inputs and image generation / editing
• Usage tracking

On the roadmap

• Text diffusion model support (Inception, Google)
• Realtime bidirectional text/audio streaming (WebRTC)

Installation

go get github.com/flitsinc/go-llms

Quick Start

Here’s a simple example that creates an LLM instance and has a conversation with it:

package main

import (
    "fmt"
    "os"

    "github.com/flitsinc/go-llms/content"
    "github.com/flitsinc/go-llms/llms"
    "github.com/flitsinc/go-llms/openai"
    "github.com/flitsinc/go-llms/tools"
)

func main() {
    // Create a new LLM instance with OpenAI's gpt-5.5 model
    llm := llms.New(
        openai.NewResponsesAPI(os.Getenv("OPENAI_API_KEY"), "gpt-5.5"),
    )

    // Optional: Set a system prompt
    llm.SystemPrompt = func() content.Content {
        return content.FromText("You are a helpful assistant.")
    }

    // Start a chat conversation
    for update := range llm.Chat("What's the capital of France?") {
        switch update := update.(type) {
        case llms.TextUpdate:
            fmt.Print(update.Text)
        }
    }
    
    // Check for errors after the chat completes
    if err := llm.Err(); err != nil {
        panic(err)
    }
}

Generating images with Gemini 3.1 Flash Image

You must specify modalities for this model to work (and you cannot use WithThinking):

func main() {
    provider := google.New("gemini-3.1-flash-image-preview").
        WithGeminiAPI(os.Getenv("GEMINI_API_KEY")).
        WithModalities("TEXT", "IMAGE")
    llm := llms.New(provider)
    for update := range llm.Chat("Draw a sketch of a happy banana") {
        switch update := update.(type) {
        case llms.TextUpdate:
            fmt.Print(update.Text)
        case llms.ImageUpdate:
            path := fmt.Sprintf("image-%d.png", time.Now().Unix())
            // We assume we always get a base64-encoded PNG image -- don't do this in prod!
            b64, _ := strings.CutPrefix(update.URL, "data:image/png;base64,")
            data, _ := base64.StdEncoding.DecodeString(b64)
            os.WriteFile(path, data, 0644)
            fmt.Printf("\nImage written to: %s\n", path)
        }
    }
}

Advanced Usage with Tools

Here’s an example showing how to use tools (function calling):

package main

import (
    "fmt"
    "os"

    "github.com/flitsinc/go-llms/anthropic"
    "github.com/flitsinc/go-llms/llms"
    "github.com/flitsinc/go-llms/tools"
)

// Define tool parameters
type CommandParams struct {
    Command string `json:"command" description:"The shell command to run"`
}

// Create a shell command tool
var RunCommand = tools.Func(
    "Run Command",
    "Run a shell command and return the output",
    "run_command",
    func(r tools.Runner, p CommandParams) tools.Result {
        return tools.SuccessWithLabel(p.Command, map[string]any{
            "output": fmt.Sprintf("Simulated output for: %s", p.Command),
        })
    },
)

func main() {
    // Create a new LLM instance using Anthropic's Claude with tools
    llm := llms.New(
        anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-6"),
        RunCommand,
    )

    // Chat with tool usage
    for update := range llm.Chat("List files in the current directory") {
        switch update := update.(type) {
        case llms.TextUpdate:
            fmt.Print(update.Text)
        case llms.ToolStartUpdate:
            fmt.Printf("(Using tool: %s)\n", update.Tool.Label()) // Shows "Run Command"
        case llms.ToolDoneUpdate:
            // Shows the label specific to this execution, e.g., "ls -l"
            fmt.Printf("(Tool result: %s)\n", update.Result.Label()) 
        }
    }
    
    // Check for errors after the chat completes
    if err := llm.Err(); err != nil {
        panic(err)
    }
}

Tools that don't exist

Models sometimes call a tool that isn't in the toolbox — a mangled name, or a real tool that was removed while the conversation history still mentions it. This doesn't end the chat: the call produces a tool "name" not found error result, which the model sees on its next turn so it can pick a different tool.

The ToolStartUpdate and ToolDoneUpdate for such a call carry a placeholder tool. Nothing is executed for it, so if you do anything at tool_start (spawn a worker, kick off a workflow), check for the placeholder first:

case llms.ToolStartUpdate:
    if tools.IsUnknown(update.Tool) {
        // There is no tool to run; the result is already an error.
        break
    }
    startExecuting(update.Tool)

Use errors.As with *tools.NotFoundError to tell this apart from an error a real tool returned.

A model that keeps calling tools that don't exist gets no rule of its own: like any other unproductive loop, it runs until WithMaxTurns stops it. That limit is unlimited by default, so set one if you don't already — otherwise a model that never stops inventing tool names never stops.

External Tools

Sometimes, you might have a set of predefined tool schemas (perhaps from an external source or another system) that you want the LLM to be able to use. AddExternalTools allows you to provide these schemas along with a single handler function.

This is useful when the logic for handling multiple tools is centralized, or when you need to dynamically add tools based on external definitions.

The handler function receives the tools.Runner and the raw JSON parameters for the called tool. You can use llms.GetToolCall(r.Context()) within the handler to retrieve the specific ToolCall instance, which includes the function name (tc.Name) and unique call ID (tc.ID), allowing you to dispatch to the correct logic.

// Example external tool schemas (could come from a config file, API, etc.)
var externalToolSchemas = []tools.FunctionSchema{
    {
        Name: "get_stock_price",
        /* ... */
    },
    {
        Name: "get_weather",
        /* ... */
    },
}

func main() {
    llm := llms.New(anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-6"))

    // Add external tools and their handler
    llm.AddExternalTools(externalToolSchemas, handleExternalTool)

    // Now the LLM can use "get_stock_price" and "get_weather"
    for update := range llm.Chat("What's the weather in London?") {
        switch update := update.(type) {
        case llms.TextUpdate:
            fmt.Print(update.Text)
        case llms.ToolStartUpdate:
            fmt.Printf("(Using tool: %s)\n", update.Tool.Label())
        case llms.ToolDoneUpdate:
            fmt.Printf("(Tool result: %s - %s)\n", update.Tool.Label(), update.Result.Label())
        }
    }
    if err := llm.Err(); err != nil {
        panic(err)
    }
}

// Single handler that forwards external tool calls.
func handleExternalTool(r tools.Runner, params json.RawMessage) tools.Result {
    // Get the specific tool call details from the context
    toolCall, ok := llms.GetToolCall(r.Context())
    if !ok {
        return tools.Errorf("Could not get tool call details from context")
    }

    // Typically, you would now:
    // 1. Construct a request to your external API endpoint (or send it to a browser client)
    targetURL := fmt.Sprintf("https://api.example.com/tool?name=%s", toolCall.Name)
    req, err := http.NewRequestWithContext(r.Context(), "POST", targetURL, bytes.NewReader(params))
    // ... set headers, handle error ...

    // 2. Execute the request.
    resp, err := httpClient.Do(req)
    // ... handle error ...

    // 3. Process the response.
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        // Use Errorf or ErrorWithLabel for tool errors
        bodyBytes, _ := io.ReadAll(resp.Body)
        return tools.Errorf("External tool API failed (%s): %s", resp.Status, string(bodyBytes))
    }
    bodyBytes, err := io.ReadAll(resp.Body)
    // ... handle read error ...

    // 4. Return the result based on the response body.
    return tools.Success(json.RawMessage(bodyBytes))
}

Grammar-Based Tools (OpenAI Only)

OpenAI supports custom tools that can enforce specific input formats using grammars. This allows you to constrain the model's output to follow precise patterns, which is useful for structured data extraction, validation, or parsing tasks.

The library supports three grammar types:

• Lark Grammar: For complex parsing using Lark parser syntax
• Regex Grammar: For pattern matching using regular expressions
• Text Grammar: For free-form text input

Here are examples of grammar-based tools:

package main

import (
    "fmt"
    "os"
    "regexp"
    "strconv"
    "strings"

    "github.com/flitsinc/go-llms/llms"
    "github.com/flitsinc/go-llms/openai"
    "github.com/flitsinc/go-llms/tools"
)

// Example 1: Lark Grammar for Math Expressions
var mathGrammar = tools.Lark(`
start: expr
expr: term (SP ADD SP term)* -> add
    | term
term: factor (SP MUL SP factor)* -> mul
    | factor
factor: INT
SP: " "
ADD: "+"
MUL: "*"
%import common.INT
`)

var MathTool = tools.FuncGrammar(
    mathGrammar,
    "Math Calculator",
    "Evaluate simple math expressions with addition and multiplication",
    "calculate_math",
    func(r tools.Runner, expression string) tools.Result {
        // Simple evaluation for demo (in practice, use a proper parser)
        expression = strings.ReplaceAll(expression, " ", "")
        
        // Handle simple cases for demonstration
        if strings.Contains(expression, "+") {
            parts := strings.Split(expression, "+")
            if len(parts) == 2 {
                a, err1 := strconv.Atoi(parts[0])
                b, err2 := strconv.Atoi(parts[1])
                if err1 == nil && err2 == nil {
                    result := a + b
                    return tools.SuccessWithLabel(expression, map[string]any{
                        "result": result,
                        "expression": expression,
                    })
                }
            }
        }
        
        // Fallback for single numbers
        if num, err := strconv.Atoi(expression); err == nil {
            return tools.SuccessWithLabel(expression, map[string]any{
                "result": num,
                "expression": expression,
            })
        }
        
        return tools.ErrorWithLabel("Invalid expression", fmt.Errorf("could not parse: %s", expression))
    },
)

// Example 2: Regex Grammar for Email Validation
var emailGrammar = tools.Regex(`^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$`)

var EmailValidator = tools.FuncGrammar(
    emailGrammar,
    "Email Validator",
    "Validate and extract information from email addresses",
    "validate_email",
    func(r tools.Runner, email string) tools.Result {
        emailRegex := regexp.MustCompile(`^([a-zA-Z0-9._%+-]+)@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,})$`)
        matches := emailRegex.FindStringSubmatch(email)
        
        if len(matches) != 3 {
            return tools.ErrorWithLabel("Invalid email", fmt.Errorf("email format is invalid: %s", email))
        }
        
        username := matches[1]
        domain := matches[2]
        
        return tools.SuccessWithLabel(email, map[string]any{
            "email": email,
            "username": username,
            "domain": domain,
            "is_valid": true,
        })
    },
)

func main() {
    // Create LLM with grammar-based tools (OpenAI only)
    llm := llms.New(
        openai.New(os.Getenv("OPENAI_API_KEY"), "gpt-5.5"),
        MathTool,
        EmailValidator,
    )

    // The model can now use structured tools with grammar constraints
    for update := range llm.Chat("Calculate 15 + 27 and validate the email user@example.com") {
        switch update := update.(type) {
        case llms.TextUpdate:
            fmt.Print(update.Text)
        case llms.ToolStartUpdate:
            fmt.Printf("(Using %s: ", update.Tool.Label())
        case llms.ToolDoneUpdate:
            fmt.Printf("%s)\n", update.Result.Label())
        }
    }
    
    if err := llm.Err(); err != nil {
        panic(err)
    }
}

Note: Grammar-based tools are currently only supported by OpenAI's API. Other providers currently do not support grammar-based tools.

Provider Support

The library currently supports:

• Anthropic
• Google (Gemini API and Vertex AI)
• OpenAI and all compatible providers (you can customize the endpoint)
• OpenAI’s newer Responses API (HTTP and WebSocket)
• TypeSafe System One models (see TypeSafe below)

Each provider can be initialized with their respective configuration:

// Anthropic
llm := llms.New(anthropic.New(os.Getenv("ANTHROPIC_API_KEY"), "claude-sonnet-4-6"))

// Google Gemini
llm := llms.New(google.New("gemini-3-flash-preview").WithGeminiAPI(os.Getenv("GEMINI_API_KEY")))

// Google Vertex AI
ts, err := googleoauth.DefaultTokenSource(ctx, "https://www.googleapis.com/auth/cloud-platform")
llm := llms.New(google.New("gemini-3-flash-preview").WithVertexAI(ts, projectID, "global"))

// OpenAI (Responses API)
llm := llms.New(openai.NewResponsesAPI(os.Getenv("OPENAI_API_KEY"), "gpt-5.5"))

// OpenAI (Responses API over WebSocket — persistent connection, faster tool loops)
provider := openai.NewWebSocketResponsesAPI(os.Getenv("OPENAI_API_KEY"), "gpt-5.5")
defer provider.Close()
llm := llms.New(provider)

// OpenAI (Chat Completions API)
llm := llms.New(openai.New(os.Getenv("OPENAI_API_KEY"), "gpt-5.5"))

// OpenAI-compatible endpoint (e.g., xAI)
// You can use the OpenAI provider with compatible APIs by configuring the endpoint.
llm := llms.New(
    openai.New(os.Getenv("XAI_API_KEY"), "grok-3-latest").
        WithEndpoint("https://api.x.ai/v1/chat/completions", "xAI"),
)

// TypeSafe (System One)
llm := llms.New(typesafe.New(os.Getenv("TYPESAFE_API_KEY"), "jev-latest"))

TypeSafe System One models

TypeSafe models such as Jev do not generate text. They read a state and answer typed questions with calibrated probabilities. The typesafe provider maps the regular go-llms call onto that API:

• The system prompt and messages become the request state, as a JSON object with a system string and a messages array. JSON content items are embedded as JSON so the model sees named fields.
• Images are lifted out of the content and sent before that object, so the state becomes an array of image parts followed by the conversation (or of the image parts alone, when the content was only images). That is the one place the API reads an image. A multimodal decision model sees the images first and the text after; a text-only model answers from the text. An image loses its place between the texts around it and its message's role, so word the text accordingly ("the screenshot shows...").
• The JSON output schema becomes the questions, one per property, with the property's description as the question. The property name is not sent to the model, so the description must carry the full meaning.
• The answers come back as the JSON object the schema describes, in one text chunk, so structured-output callers work unchanged.

Some System One models refuse that conversation object and read only a state of their own shape; OpenRouter's Respan Span-01, for example, takes a transcript whose last assistant message is judged. WithState replaces the default state with whatever a builder returns from the system prompt and messages. The builder owns the conversion, including tool messages, which reach it instead of being refused, and its error fails the request before anything is sent.

Supported property types:

Property type	Question	Rendered answer
boolean	Noul	true when the probability is at least 0.5
number	Noul	the probability that the description holds, 0 to 1
string with enum	Choice	the chosen enum member

A number property is read as the probability that its description holds, so its description has to be a yes/no question. The rendered value is that probability, not a quantity the model counted or estimated.

A property listed in the schema's required must be answered or the request fails; an unanswered optional property is simply left out of the rendered object.

Nested objects, arrays, free strings, and tools are rejected with a typed error rather than approximated, because the model cannot produce them; audio, video, and compaction content are rejected because the state cannot hold them. Callers that need the full probability distribution or confidence behind an answer can read it from the stream. That means calling the provider's Generate directly, since llms.LLM does not expose provider streams:

provider := typesafe.New(os.Getenv("TYPESAFE_API_KEY"), "jev-latest")
stream := provider.Generate(ctx, systemPrompt, messages, nil, schema)
if err := stream.Err(); err != nil {
    panic(err)
}
if response := stream.(*typesafe.Stream).Response(); response != nil {
    fmt.Println(response.Answers["severity"].Probabilities)
}

WebSocket Provider (OpenAI)

The WebSocket provider keeps a persistent connection to OpenAI's Responses API. Within multi-turn tool loops it automatically chains responses via previous_response_id, sending only new tool results instead of the full conversation history. This can significantly reduce latency in tool-heavy workflows.

provider := openai.NewWebSocketResponsesAPI(os.Getenv("OPENAI_API_KEY"), "gpt-5.5")
defer provider.Close() // closes the WebSocket when done

The connection is established lazily on the first call. All the same With* configuration methods from ResponsesAPI are available.

Warmup pre-loads tools and instructions on the server before the first user message, making the first turn faster:

_, err := provider.Warmup(ctx, "You are a helpful assistant.", toolbox)

Debugging — when calling Warmup or Generate directly on the provider (not through LLM), use WithDebugger to attach a debugger:

provider.WithDebugger(llms.StdOutDebugger)

Alternatively, pass the debugger via context: llms.WithDebugger(ctx, llms.StdOutDebugger).

External connections let you manage the WebSocket lifecycle yourself:

conn, err := openai.DialResponsesWebSocket(ctx, accessToken)
provider := openai.NewWebSocketResponsesAPI(accessToken, "gpt-5.5").WithConn(conn)
// provider.Close() is a no-op — you own the connection

ResetChain clears the chaining state when starting a new conversation on the same connection:

provider.ResetChain()

You can easily implement new providers by implementing the Provider interface:

type Provider interface {
    Company() string
    Model() string
    // Generate takes a system prompt, message history, and optional toolbox,
    // returning a stream for the LLM's response. The provided context should
    // be respected for cancellation.
    //
    // Providers should check for a Debugger on the context via
    // llms.GetDebugger(ctx) to support request/response logging.
    Generate(
        ctx context.Context,
        systemPrompt content.Content,
        messages []Message,
        toolbox *tools.Toolbox,
        jsonOutputSchema *tools.ValueSchema,
    ) ProviderStream
}

type ProviderStream interface {
    Err() error
    Iter() func(yield func(StreamStatus) bool)
    Message() Message
    Text() string
    Image() (string, string)
    Audio() (string, string)
    Thought() content.Thought
    ToolCall() ToolCall
    Usage() Usage
}

A stream can also implement optional capabilities, which the turn loop discovers with a type assertion: ContextUsageStream (usage of the final sampling pass, after any compaction), SearchStream (provider-run searches), CompactionStream (native compaction checkpoints) and ToolArgumentFinalizationStream (a final snapshot of tool arguments).

To wrap a stream (for fallback, metering and the like), embed llms.StreamWrapper rather than llms.ProviderStream, so the wrapped stream's optional capabilities keep reaching the turn loop, and assert llms.StreamCapabilities to catch a regression at compile time. A wrapper that overrides Usage must override ContextUsage too:

type meteredStream struct {
    llms.StreamWrapper
}

var _ llms.StreamCapabilities = (*meteredStream)(nil)

stream := &meteredStream{StreamWrapper: llms.StreamWrapper{ProviderStream: inner}}

Usage Tracking

Track the usage of your LLM interactions:

usage := llm.TotalUsage
fmt.Printf("Cached Input Tokens: %d, Tokens Written to Input Cache: %d, Input Tokens: %d, Output Tokens: %d\n",
    usage.CachedInputTokens, usage.CacheCreationInputTokens, usage.InputTokens, usage.OutputTokens)

As patterns emerge between providers with regards to cache tokens, speculative tokens, etc. these will be added too.

When to use this?

When you want to make providers easily swappable and a simplified API that focuses on hekoing you implement the most common types of agentic flows.

Since each LLM provider has its own quirks, especially around reasoning, streaming, and tool calling, we’ve done our best to smooth those over, but expect some differences still.

Provider quirks

additionalProperties forbidden and required depending on provider

Google doesn’t allow the additionalProperties field for JSON schemas (probably a bug), while OpenAI’s new Responses API requires it for tool calls! It’s also commonly required for models with strict JSON outputs since it helps with speculative decoding.

Because of this, we strip out the additionalProperties field before sending it to Google, so it shouldn’t be a problem for you, just keep it in mind.

Optional properties under OpenAI strict mode

OpenAI’s strict mode (Responses API function tools, and JSON outputs on both OpenAI APIs) rejects a schema unless every property is listed in required, so optional properties have to be sent as required-but-nullable (anyOf: [<schema>, {"type": "null"}]). The types that carry the strict flag pad their schema when they are encoded, including a FunctionTool you pass to WithTool, so write optional properties normally and expect null for the ones the model leaves out on those requests (tools.Func validation treats null for an optional property as left out). An object whose additionalProperties is the any-value schema {} is closed (false), because strict mode needs a type on every schema; strict mode also needs additionalProperties: false on every object, which padding does not add, so close your objects yourself. Padding is idempotent, so schemas you already pad yourself are sent unchanged.

Everywhere else the schema goes out as written, including Chat Completions function tools, which are not sent strict unless you call WithStrictTools(). Do call it for Chat Completions endpoints that serve OpenAI models (including OpenRouter’s openai/* models): OpenAI treats function tools as strict even without the flag, so an unpadded schema makes the model invent a value for every optional field. Don’t pad schemas yourself for other providers: forced to fill an optional field it wants to leave out, Claude sometimes writes an empty value, which Anthropic streams as invalid argument JSON ("field": ,).

Array length limits under Anthropic structured outputs

Anthropic’s structured outputs reject maxItems and any minItems other than 0 or 1, so the anthropic package drops maxItems from a JSON output schema and lowers a larger minItems to 1. Tool schemas go out with their limits, which Claude reads as hints. Other providers receive the limits as written but enforce them unevenly, so also state a limit in the field description or the prompt: schema enforcement is a backstop, not a replacement.

OpenAI reasoning belongs to an organization

OpenAI resolves a reasoning item replayed by ID alone only in the organization that stored it, so that history fails once the API key belongs to another organization. The Responses API provider therefore asks for each reasoning item’s encrypted content and keeps it on the content.Thought (Encrypted, next to the item’s ID), both in the streamed thought and in the message. Only the endpoint that produced the content can read it, so the thought’s Metadata records its openai:reasoning_format (openai-responses-v1) and openai:reasoning_endpoint (the API host, such as api.openai.com or api.x.ai). The Responses provider replays the content only to that endpoint and replays other thoughts by ID, and Chat Completions forwards the format so OpenRouter can drop reasoning its upstream cannot read. Keep Metadata when you persist thoughts.

A thought recorded before this content was requested has only its ID and can still be refused after a change of organization, as can content OpenAI can no longer verify. (*llms.HTTPError).IsReplayRejected() reports these refusals, and those of compaction checkpoints; the caller should then send the conversation without the replayed reasoning and checkpoints. The WebSocket provider resends the whole conversation once, without previous_response_id, when the server no longer holds the response it chained to.

Anthropic doesn’t stream partial property values by default

The streaming API of Anthropic only sends complete string values when streaming tool calls, so if you have a tool call like edit_file which produces very long fields nothing will update until that field has completely finished generating.

To fix this, use fine-grained tool streaming which is currently in beta, by calling .WithBeta("fine-grained-tool-streaming-2025-05-14") on the Anthropic provider instance.

License

MIT License - See LICENSE file for details.