bedrock

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2026 License: MIT Imports: 11 Imported by: 0

README

AWS Bedrock LLM Provider

Production-ready Go client for AWS Bedrock with support for 40+ models across 11 providers (Amazon, Anthropic, Meta, DeepSeek, OpenAI, Qwen, Mistral, Moonshot, Z.AI, MiniMax, NVIDIA).

Architecture Overview

graph TB
    Client[Client Code] --> LLM[bedrock.LLM]
    LLM --> Processing[Message Processing]
    Processing --> Legacy[Legacy API Client]
    Processing --> Converse[Converse API Client]
    Legacy --> Anthropic[provider_anthropic.go]
    Legacy --> Nova[provider_nova.go]
    Legacy --> Others[provider_*.go]
    Converse --> ConverseClient[ConverseClient]
    Anthropic --> AWS[AWS SDK]
    Nova --> AWS
    Others --> AWS
    ConverseClient --> AWS
Three-Layer Design

Why this architecture?

  1. Public API Layer (bedrockllm.go)

    • Single entry point: bedrock.LLM
    • Hides complexity of dual API support
    • Manages automatic caching logic
  2. Message Processing Layer (processMessages())

    • Converts llms.MessageContentbedrockclient.Message
    • Handles tool calls, reasoning, multimodal content
    • Applies automatic cache control insertion
  3. Internal Client Layer (internal/bedrockclient/)

    • Provider-specific implementations
    • AWS SDK interaction
    • Response parsing and streaming

Dual API Strategy

Why Two APIs?

Legacy API (InvokeModel/InvokeModelWithResponseStream):

  • Direct access to model-specific features
  • Anthropic cache_control format support
  • Broader model compatibility
  • Use when: Model not supported by Converse API

Converse API (Converse/ConverseStream):

  • Unified interface across all models
  • Native cachePoint support (AWS SDK types)
  • Better error handling
  • Use when: Building new applications (recommended)

Decision Point: useConverseAPI flag in LLM struct

flowchart LR
    A[GenerateContent] --> B{useConverseAPI?}
    B -->|true| C[generateContentWithConverseAPI]
    B -->|false| D[generateContentWithLegacyAPI]
    C --> E[ConverseClient]
    D --> F[Provider Detection]
    F --> G[createAnthropicCompletion]
    F --> H[createNovaCompletion]
    F --> I[create*Completion]

Automatic Prompt Caching

How It Works

Challenge: Anthropic's prompt caching requires manual cache control wrappers on client side.

Solution: Automatic cache point insertion for Claude 4.x models.

sequenceDiagram
    participant Client
    participant LLM
    participant Processing
    participant API

    Client->>LLM: GenerateContent(messages)
    LLM->>LLM: supportsCaching(modelID)?
    alt Claude 4.x model
        LLM->>Processing: processMessagesWithCaching(autoCaching=true)
        Processing->>Processing: applyAutomaticCaching()
        Note over Processing: Add cache control to<br/>last assistant/tool message
    else Other model
        LLM->>Processing: processMessagesWithCaching(autoCaching=false)
    end
    Processing->>API: Send with cache points
    API-->>LLM: Response with cache metrics

Implementation:

  • supportsCaching(): Pattern matching on model ID (claude-opus-4, claude-sonnet-4, claude-haiku-4)
  • applyAutomaticCaching(): Adds CacheControl{Type: "ephemeral", TTL: "5m"} to last cacheable message (assistant or tool response)
  • Why last message? Caches conversation history before new user input
  • TTL Options: 5 minutes (default) or 1 hour (configurable via EphemeralCacheOneHour())

Benefits:

  • 90% cost reduction on cached tokens
  • Zero client code changes with automatic caching
  • Works with both Legacy and Converse APIs
  • Supports both manual (WithCacheControl()) and automatic (WithAutomaticCaching()) modes

Tool Calling Implementation

Provider-Specific Differences

Why different implementations?

Different providers use different formats for tool inputs in Converse API:

Provider Input Format Handled By
Anthropic Native JSON objects map[string]any direct
Nova Native JSON objects map[string]any direct
Qwen / Mistral / MiniMax / Nemotron Native JSON objects map[string]any direct
GLM (Z.AI) String-based Not supported

Problems with specific models:

  • GLM (Z.AI): Backend expects tool input as string, not JSON object. This breaks Converse API spec.
  • Meta Llama 3.3/3.1 70B/8B: Unstable behavior when processing tool call results.
  • Mistral Magistral Small: Tool calling not supported.
  • Moonshot Kimi K2-Thinking: Unstable tool calling behavior in streaming mode.
  • Qwen3-VL: Unstable tool calling in streaming mode.

Solution: Models with backend issues or instability are excluded from tool calling tests.

convertToolCallInput() Logic

Why this function exists?

AWS SDK requires Smithy-compatible types for document.NewLazyDocument(). Standard Go types from json.Unmarshal aren't always compatible.

func convertToolCallInput(args any) (any, error) {
    // 1. Check Smithy compatibility (structs with `document` tags)
    if isSmithyValidObject(args) {
        return args, nil
    }
    
    // 2. Re-encode to normalize types for Smithy SDK
    // Uses UseNumber() to preserve numeric precision during JSON roundtrip
    jsonBytes := bytes.NewBuffer(nil)
    json.NewEncoder(jsonBytes).Encode(args)
    
    decoder := json.NewDecoder(jsonBytes)
    decoder.UseNumber()  // Preserves large numbers accurately
    
    var jsonValue any
    decoder.Decode(&jsonValue)
    return jsonValue, nil
}

Why UseNumber()?

Preserves numeric precision for large integers that might overflow float64. The json.Number type is properly handled by Smithy SDK's document encoding.

Document Marshal for Tool Responses

Why MarshalSmithyDocument()?

When extracting tool call from response, block.Value.Input is document.LazyDocument, not plain Go type:

// WRONG: json.Marshal on document - loses type info
argsJSON, _ := json.Marshal(block.Value.Input)  

// CORRECT: Use MarshalSmithyDocument to get JSON bytes directly
argsJSON, err := block.Value.Input.MarshalSmithyDocument()
if err != nil {
    return fmt.Errorf("failed to marshal tool input: %w", err)
}
// argsJSON is now []byte with the JSON representation

Reasoning (Thinking) Support

Models Supporting Reasoning

Converse API:

  • Claude: Fable 5, Opus 5/4.8/4.7/4.6/4.5, Sonnet 5/4.6/4.5, Haiku 4.5
  • DeepSeek R1
  • OpenAI GPT OSS (120B, 20B)
  • Moonshot Kimi K2-Thinking

Legacy API (InvokeModel):

  • Claude: Opus 5/4.8/4.7/4.6/4.5, Sonnet 5/4.6/4.5, Haiku 4.5

How the wire shape is resolved

Claude thinking is not a single flag — the generation resolves adaptive vs. budget thinking from the model via the shared llms/reasoning capability tables (the same source of truth used by the first-party Anthropic provider):

  • Adaptive-only (Opus 4.7/4.8/5, Sonnet 5, Fable 5): thinking.type=adaptive + output_config.effort; budget thinking and sampling params are rejected. Opus 5, Sonnet 5, and Fable 5 think by default (Opus 5 is a breaking change from Opus 4.8, which defaults off); on Bedrock a default-on model cannot be explicitly disabled (always-on there), while Opus 4.7/4.8 default off, so omitting thinking already yields off.
  • Adaptive + budget (Opus 4.6, Sonnet 4.6): either mechanism; caller preference honored.
  • Budget-only (Opus 4.5, Sonnet 4.5, Haiku 4.5): thinking.type=enabled + budget_tokens; Opus 4.5 also honors output_config.effort.

Non-Claude reasoning models (DeepSeek R1, GPT OSS, Kimi K2-Thinking) use budget thinking through the Converse API. WithReasoningDisabled() returns a typed ErrReasoningOffUnsupported for always-on Bedrock models.

Structured Output

The provider-neutral llms.WithStructuredOutput is supported on both API paths for Anthropic models: the final response is guaranteed to be a single JSON value matching the supplied JSON Schema (Draft 2020-12), validated locally against the original schema.

schema := json.RawMessage(`{
    "type": "object",
    "properties": {"country": {"type": "string"}, "capital": {"type": "string"}},
    "required": ["country", "capital"],
    "additionalProperties": false
}`)

resp, err := llm.GenerateContent(ctx, messages,
    llms.WithStructuredOutput(llms.StructuredOutputConfig{Name: "capital", Schema: schema}))

Wire mapping:

  • Converse: native OutputConfig.TextFormat with a JsonSchemaDefinition (AWS SDK types). Rides both Converse and ConverseStream.
  • Legacy (InvokeModel): Anthropic-compatible output_config.format, merged with reasoning output_config.effort when both are set.

Requirements and behavior:

  • Every object node must set additionalProperties: false — Bedrock rejects a schema that omits it. The SDK enforces this locally with a typed ErrStructuredOutputConfig before the request is sent.
  • Only Anthropic models are supported on the legacy path; a non-Anthropic legacy model returns a typed unsupported-path error. Converse is not restricted to Claude — any model AWS advertises as supporting Structured Outputs works.
  • Only the final normal turn (end_turn/stop_sequence) is validated; a tool_use/max_tokens/guardrail/filtered turn is not treated as final JSON.
  • The response StopReason is surfaced on ContentChoice.StopReason (Converse now transfers it from the response/MessageStopEvent).

Why these models?

Extended thinking/reasoning capabilities are model-specific features. DeepSeek R1, OpenAI OSS, and Moonshot models provide reasoning through the Converse API, while Anthropic models support both APIs.

Message Structure with Reasoning
flowchart TB
    A[AI Message with Reasoning] --> B{Legacy or Converse?}
    B -->|Legacy API| C[Add thinking blocks<br/>BEFORE text content]
    B -->|Converse API| D[Add ReasoningContent blocks<br/>with signature]
    C --> E[anthropicTextGenerationInputContent array]
    D --> F[types.ContentBlock array]

Why order matters?

Anthropic API spec requires thinking blocks before text blocks in assistant messages.

Signature Preservation

Challenge: Reasoning signatures must round-trip through conversations.

Solution: Store in reasoning.ContentReasoning.Signature field, re-insert on next turn.

// Receive
choice.Reasoning.Signature = []byte(...)

// Send back
llms.TextPartWithReasoning(content, reasoning)

Message Processing Pipeline

flowchart LR
    A[llms.MessageContent] --> B[processMessagesWithCaching]
    B --> C{autoCaching?}
    C -->|true| D[applyAutomaticCaching]
    C -->|false| E[bedrockclient.Message array]
    D --> E
    E --> F{API Type}
    F -->|Legacy| G[Provider-specific format]
    F -->|Converse| H[types.Message]
    G --> I[InvokeModel]
    H --> J[Converse]
Key Transformations

llms.MessageContent → bedrockclient.Message:

  • Role mapping (System/Human/AI/Tool → provider format)
  • Content type detection (Text/Binary/ToolCall/ToolResponse)
  • Reasoning extraction and formatting
  • Cache control attachment

bedrockclient.Message → AWS Request:

  • Legacy: JSON serialization with provider schemas
  • Converse: AWS SDK types with document encoding

Adding New Models

Step-by-Step Process
  1. Add Model Constant (models_list.go)
// Include: Description, max tokens, languages, use cases
ModelNewProvider = "provider.model-id"
  1. Update Provider Detection (if new provider)
// bedrockclient.go
case strings.Contains(modelID, "newprovider"):
    return "newprovider"
  1. Implement Provider (internal/bedrockclient/provider_new.go)
func createNewProviderCompletion(ctx, client, modelID, messages, options) {
    // 1. Convert messages to provider format
    // 2. Build request payload
    // 3. Call InvokeModel
    // 4. Parse response
    // 5. Return llms.ContentResponse
}
  1. Register in Switch (bedrockclient.go)
case "newprovider":
    return createNewProviderCompletion(...)
  1. Add Tests (bedrockllm_test.go)
  • Add model to TestAmazonOutputConverseAPI models list (if Converse API supported)
  • Add model to TestAmazonOutputLegacyAPI models list (Legacy API)
  • Add model to TestAmazonStreamingOutputConverseAPI (if streaming supported)
  • Add model to TestAmazonStreamingOutputLegacyAPI (if streaming with Legacy API supported)
  • Add model to TestAmazonToolCallingConverseAPI (if tools supported)
  • Add model to TestAmazonToolCallingLegacyAPI (if tools with Legacy API supported)
  • Add model to TestAmazonReasoningConverseAPI (if reasoning/thinking supported)
  • Add model to TestAmazonReasoningLegacyAPI (if reasoning with Legacy API supported)
  • Structured output is covered by TestAmazonStructuredOutputConverseAPI and TestAmazonStructuredOutputStreamingConverseAPI (Converse); schemas must set additionalProperties:false
  1. Record HTTP Interactions
HTTPRR_RECORD=. go test -v -run TestNewModel
Provider Implementation Checklist
  • Input struct with all parameters (Temperature, TopP, MaxTokens, etc.)
  • Output struct matching API response
  • Streaming struct if model supports streaming
  • Error handling for provider-specific errors
  • Token usage extraction
  • Stop reason mapping

Testing Strategy

Why httprr?

Problem: Integration tests require AWS credentials and cost money.

Solution: Record HTTP interactions once, replay for fast tests.

# Record new interactions
HTTPRR_RECORD=. go test -v -run TestName

# Debug during recording
HTTPRR_RECORD=. HTTPRR_DEBUG=true go test -v -run TestName

# Replay (default)
go test -v -run TestName
Test Organization

Test File:

  1. bedrockllm_test.go: Integration tests (requires AWS credentials)

    • Model output validation (Converse and Legacy API)
    • Streaming behavior (Converse and Legacy API)
    • Tool calling workflows (Converse and Legacy API, with streaming variants)
    • Reasoning roundtrips (Converse and Legacy API, with streaming variants)
    • Caching metrics (automatic and manual caching)
    • Extended thinking with tool calls
    • Multi-turn caching with tools
    • Client creation with different credential types (long-lived, bearer token)
  2. bedrockllm_unit_test.go: Unit tests (no credentials)

    • Message processing logic
    • Option configuration
    • Cache control application
    • Provider detection

Why separate files?

  • Unit tests run in CI without credentials
  • Integration tests record once, replay forever
  • Tool tests validate complex workflows
Testing Non-Deterministic Tool Calls

Challenge: map[string]any serialization order is non-deterministic.

// First run: {"a": 15, "b": 8}
// Second run: {"b": 8, "a": 15}  // Different order!

Solution: Skip second request in replay mode (isReplaying check).

if isReplaying {
    return nil  // Don't send tool result
}

Error Handling

Error Mapping Strategy

Why custom mapping?

AWS errors are provider-specific strings. Need standardized codes for client logic.

// errors.go
bedrockErrorMappings = []errorMapping{
    {patterns: []string{"throttlingexception"}, code: llms.ErrCodeRateLimit},
    {patterns: []string{"accessdenied"}, code: llms.ErrCodeAuthentication},
    // ...
}

Usage:

if llmErr, ok := err.(*llms.Error); ok {
    switch llmErr.Code {
    case llms.ErrCodeRateLimit:
        // Implement backoff
    }
}

Streaming Implementation

Event Processing Pattern

Both APIs use AWS SDK event streams, but different event types:

Legacy API:

// Anthropic: streamingCompletionResponseChunk
types: message_start, content_block_delta, message_delta, message_stop

Converse API:

// AWS SDK types
ConverseStreamOutputMemberContentBlockDelta
ConverseStreamOutputMemberContentBlockStart
ConverseStreamOutputMemberContentBlockStop
Tool Call Streaming Accumulation

Why accumulation needed?

Tool arguments arrive in chunks:

// Chunk 1: {"operation"
// Chunk 2: :"multiply","a"
// Chunk 3: :15,"b":8}

Solution: Accumulate in streaming.ToolCall map, send complete call at ContentBlockStop.

Maintenance Guidelines

When to Use Legacy vs Converse API

Use Legacy API:

  • Model doesn't support Converse API
  • Need Anthropic-specific cache_control format
  • Debugging provider-specific issues

Use Converse API:

  • Default for new implementations
  • Better error messages
  • Unified tool calling
  • Native cachePoint support
Adding Caching Support

Criteria:

  1. Model must support Anthropic prompt caching (currently only Claude 4.x)
  2. Add pattern to supportsCaching() in bedrockllm.go:
cachingPatterns := []string{
    "claude-opus-4",
    "claude-sonnet-4",
    "claude-haiku-4",
    "claude-new-4",  // Add new model pattern
}
  1. Ensure model supports minimum 1024 tokens threshold for cache activation
  2. Add tests in TestAmazonAutomaticCachingConverseAPI and TestAmazonAutomaticCachingLegacyAPI
Common Pitfalls

1. Forgetting to handle nil FunctionCall

// BAD
Name: part.FunctionCall.Name  // Panic if nil

// GOOD
if part.FunctionCall == nil {
    return errors.New("missing function call")
}

2. Ignoring marshal errors

// BAD
argsJSON, _ := json.Marshal(data)

// GOOD
argsJSON, err := json.Marshal(data)
if err != nil {
    return fmt.Errorf("marshal failed: %w", err)
}

3. Not checking tool call arguments validity

// BAD - may pass invalid JSON to tool
toolCall.FunctionCall.Arguments  // No validation

// GOOD - validate tool arguments can be parsed
var args map[string]any
if err := json.Unmarshal([]byte(toolCall.FunctionCall.Arguments), &args); err != nil {
    return fmt.Errorf("invalid tool arguments: %w", err)
}

File Organization

llms/bedrock/
├── bedrockllm.go              # Main LLM implementation, message processing, caching
├── bedrockllm_option.go       # Configuration options
├── bedrockllm_test.go         # Integration tests (httprr recorded)
├── bedrockllm_unit_test.go    # Unit tests (no AWS required)
├── models_list.go             # Model constants and documentation
├── errors.go                  # Error mapping
├── doc.go                     # Package documentation
├── tool_call_test.go          # Tool call processing tests
├── llmtest_test.go            # LLM interface compliance tests
└── internal/bedrockclient/
    ├── bedrockclient.go           # Legacy API client
    ├── bedrockclient_converse.go  # Converse API client
    ├── bedrockclient_util.go      # Smithy validation
    ├── bedrockclient_test.go      # Client tests
    ├── bedrockclient_integration_test.go  # Client integration tests
    ├── provider_anthropic.go      # Anthropic-specific implementation
    ├── provider_nova.go           # Nova-specific implementation
    ├── provider_*.go              # Other providers
    └── *_test.go                  # Provider tests

Key Design Decisions

Why Separate Provider Files?

Problem: Single file would be 5000+ lines with mixed concerns.

Solution: Each provider in separate file with consistent interface.

Benefits:

  • Easy to add new providers
  • Clear separation of model-specific logic
  • Isolated testing
Why Two Cache Control Formats?

Legacy API: Anthropic's cache_control in message content

{
  "type": "text",
  "text": "...",
  "cache_control": {"type": "ephemeral", "ttl": "5m"}
}

Converse API: AWS's cachePoint blocks

{
  "content": [
    {"text": "..."},
    {"cachePoint": {"type": "default", "ttl": "fiveMinutes"}}
  ]
}

Why support both?

  • Legacy API: Anthropic cache_control format required (embedded in content blocks)
  • Converse API: AWS cachePoint format required (separate content block type)
  • Automatic caching abstracts the difference by using appropriate format based on API
Message Processing: Content Parts to Messages

Key Insight: llms.MessageContent supports multiple content parts (text, images, tool calls), but provider APIs may require them as separate messages or flattened arrays.

Processing Flow:

// Input: Single MessageContent with multiple parts
llms.MessageContent{
    Role: llms.ChatMessageTypeHuman,
    Parts: [TextPart("Describe"), BinaryPart(imageData)]
}

// Output: Flat message array for provider API
[
    Message{Type: "text", Content: "Describe"},
    Message{Type: "image", Content: base64Data, MimeType: "image/jpeg"}
]

Why flatten?

  • Anthropic Legacy API: Requires flat content array per message
  • Nova/other providers: Each content type is separate array element
  • Simplifies provider-specific serialization logic

Debugging Guide

Enable HTTP Logging
HTTPRR_DEBUG=true go test -v -run TestName
Check Cache Metrics
resp.Choices[0].GenerationInfo["CacheReadInputTokens"]      // Tokens read from cache
resp.Choices[0].GenerationInfo["CacheCreationInputTokens"]  // Tokens written to cache
Validate Tool Call Arguments

Look for empty Arguments in logs - indicates parsing issue:

if len(arguments) == 0 {
    // Check FunctionCall.Arguments JSON validity
}
Common Errors

"unsupported message type"

  • Added new ContentPart type without handling in processMessages()

"role not supported"

  • Provider doesn't support message role (e.g., Function role)

"completed due to max_tokens"

  • Increase MaxTokens in request

"cached HTTP response not found"

  • httprr recording changed, re-record with HTTPRR_RECORD=.

Performance Considerations

Automatic Caching Impact

First request: Cache creation overhead (~50ms) Subsequent requests: 90% token cost reduction, ~20% latency reduction

Best for:

  • Long conversations (3+ turns)
  • Large system prompts (>1024 tokens)
  • Repeated context (tools, RAG documents)
Streaming Latency

Time to first token: 200-500ms (depending on model) Chunk frequency: Every 20-50ms

Use streaming when:

  • User-facing chat interfaces
  • Long responses (>500 tokens)
  • Real-time feedback needed

Future Enhancements

Potential Improvements
  1. Converse API Migration

    • Move all models to Converse API
    • Deprecate Legacy API providers
    • Why: Simpler maintenance, better consistency
  2. Smart Cache TTL Selection

    • 5m for short conversations
    • 1h for long sessions
    • Why: Optimize cost vs cache hit rate
  3. Parallel Tool Calls

    • Support multiple simultaneous tools
    • Why: Some models return parallel tool calls
  4. Structured Output for non-Anthropic legacy models

    • The legacy InvokeModel structured-output path is Anthropic-only; other providers currently return a typed unsupported-path error (use Converse)
    • Why: each legacy provider needs its own request shape

(Schema-constrained structured output itself is already implemented — see the Structured Output section.)

Contributing

Before Adding Features
  1. Check if Converse API supports it natively
  2. Consider impact on both API paths
  3. Add tests for both streaming and non-streaming
  4. Update httprr recordings
Code Review Checklist
  • Error handling for all AWS SDK calls
  • Nil checks for optional fields
  • Cache control doesn't break non-caching models
  • Tool calling tested with real model
  • httprr recordings committed
  • Documentation updated

Supported Model Matrix

Structured Output and Caching apply to Anthropic (Claude) models. See models_list.go for the exact model IDs.

Provider Tool Calling Reasoning Streaming Multimodal Caching Structured Output
Claude Fable 5 ✅ (always-on)
Claude Opus 5/4.8/4.7/4.6/4.5
Claude Sonnet 5/4.6/4.5
Claude Haiku 4.5
Nova 2/Pro/Lite/Micro Converse native*
Llama 4 / 3.x Limited Converse native*
DeepSeek V3.2 Converse native*
DeepSeek R1 Converse native*
OpenAI GPT (OSS) Converse native*
Qwen3 Varies** Some Converse native*
Mistral ✅*** Some Converse native*
Moonshot Kimi ✅**** Some Converse native*
MiniMax M2/M2.1/M2.5 Converse native*
GLM-4.7/4.7-Flash/5 ❌***** ✅ (GLM-5) Converse native*
NVIDIA Nemotron 3 Super Converse native*

*Converse native: structured output is passed via AWS OutputConfig.TextFormat; support depends on what AWS advertises for the model at the time. The legacy InvokeModel structured-output path is implemented for Anthropic only.
**Qwen3: Most models support tools, except Qwen3-VL (unstable in streaming)
***Mistral: Large 3 and Large 2402 support tools, Magistral Small 2509 does not
****Moonshot: K2.5 supports tools, K2-Thinking is unstable in streaming
*****GLM models: Backend incompatibility with Converse API tool format (requires string instead of JSON)

See models_list.go for the complete model list and detailed capabilities.

Documentation

Overview

Package bedrock provides AWS Bedrock integration for LangChainGo.

Overview

This package implements LLM client for AWS Bedrock, supporting multiple model providers including Anthropic Claude, Amazon Nova, Meta Llama, Cohere, AI21, and DeepSeek.

Architecture

The package consists of three layers:

  1. Public API Layer (bedrockllm.go): Exposes bedrock.LLM and bedrock.New() constructor
  2. Message Processing Layer: Converts llms.MessageContent to provider-specific formats
  3. Internal Client Layer (internal/bedrockclient): Handles AWS SDK interactions

Two API modes are supported:

  • Legacy API: Model-specific implementations via InvokeModel/InvokeModelWithResponseStream
  • Converse API: Unified implementation via Converse/ConverseStream (recommended)

Basic Usage

Create a Bedrock client:

import "github.com/nvroot/langchaingo/llms/bedrock"

llm, err := bedrock.New(
    bedrock.WithModel(bedrock.ModelAnthropicClaudeSonnet45),
    bedrock.WithConverseAPI(),
)

Generate content:

messages := []llms.MessageContent{
    llms.TextParts(llms.ChatMessageTypeHuman, "Hello!"),
}
resp, err := llm.GenerateContent(ctx, messages,
    llms.WithMaxTokens(1024),
)

Automatic Prompt Caching

For Claude 4.x models (Opus 4, Sonnet 4, Haiku 4), automatic caching is available:

llm, err := bedrock.New(
    bedrock.WithModel(bedrock.ModelAnthropicClaudeSonnet45),
    bedrock.WithConverseAPI(),
    bedrock.WithAutomaticCaching(),  // Enable automatic caching
)

When enabled, the client automatically:

  • Detects Claude 4.x models by model ID patterns (anthropic.claude-opus-4, sonnet-4, haiku-4)
  • Adds cache points to the last assistant or tool message before new user input
  • Uses ephemeral 5-minute TTL by default
  • Works transparently without modifying client code

Benefits:

  • 90% cost reduction on cached input tokens
  • No manual cache control wrappers needed
  • Automatic conversation history caching

Manual caching (for fine-grained control):

messages := []llms.MessageContent{
    {
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            bedrock.WithCacheControl(
                llms.TextPart("long context..."),
                bedrock.EphemeralCache(),
            ),
        },
    },
}

Tool Calling

Both APIs support tool calling for compatible models:

tools := []llms.Tool{
    {
        Type: "function",
        Function: &llms.FunctionDefinition{
            Name: "get_weather",
            Description: "Get weather for location",
            Parameters: map[string]any{...},
        },
    },
}

resp, err := llm.GenerateContent(ctx, messages,
    llms.WithTools(tools),
)

Reasoning Support

Claude 4.x and 3.7 models support reasoning (thinking) mode:

resp, err := llm.GenerateContent(ctx, messages,
    llms.WithReasoning(llms.ReasoningMedium, 2048),
)

// Access reasoning content
if resp.Choices[0].Reasoning != nil {
    fmt.Println(resp.Choices[0].Reasoning.Content)
}

Streaming

Both APIs support streaming responses:

streamFunc := func(ctx context.Context, chunk streaming.Chunk) error {
    switch chunk.Type {
    case streaming.ChunkTypeText:
        fmt.Print(chunk.Content)
    case streaming.ChunkTypeReasoning:
        fmt.Println("Thinking:", chunk.Reasoning.Content)
    case streaming.ChunkTypeToolCall:
        fmt.Println("Tool:", chunk.ToolCall.Name)
    }
    return nil
}

resp, err := llm.GenerateContent(ctx, messages,
    llms.WithStreamingFunc(streamFunc),
)

Supported Models

See models_list.go for complete list. Major providers:

  • Anthropic: Claude 4.6 (Opus, Sonnet), Claude 4.5, 4.1, 4, 3.7, 3.5
  • Amazon: Nova 2 Lite, Nova Premier, Nova Pro, Nova Lite, Nova Micro
  • Meta: Llama 4, Llama 3.3, 3.2, 3.1, 3
  • Cohere: Command R, Command R+
  • AI21: Jamba 1.5 Large, Mini
  • DeepSeek: R1
  • OpenAI: GPT-OSS-120B, GPT-OSS-20B
  • Qwen: Qwen3 Next, Qwen3 VL, Qwen3 32B, Qwen3 Coder (30B, Next)
  • Mistral: Large 3, Magistral Small
  • Moonshot: Kimi K2.5, Kimi K2 Thinking
  • Z.AI: GLM-4.7, GLM-4.7-Flash

Error Handling

Provider-specific errors are mapped to standardized error codes:

resp, err := llm.GenerateContent(ctx, messages)
if err != nil {
    if llmErr, ok := err.(*llms.Error); ok {
        switch llmErr.Code {
        case llms.ErrCodeRateLimit:
            // Handle rate limiting
        case llms.ErrCodeAuthentication:
            // Handle auth errors
        }
    }
}

See errors.go for complete error mapping.

AWS Configuration

The client uses AWS SDK v2 configuration:

  • Credentials: From environment (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) or AWS config
  • Region: From environment (AWS_REGION) or default config
  • Custom configuration: Use bedrock.WithClient() with pre-configured bedrockruntime.Client

Performance Considerations

  • Converse API is recommended for new applications (unified, better error handling)
  • Automatic caching reduces costs by 90% for cached tokens (Claude 4.x only)
  • Streaming reduces latency for interactive applications
  • Minimum cache checkpoint: 1024 tokens (Sonnet 4.5), 4096 tokens (Haiku 4.5)

Maintenance

When adding new models:

  1. Add model constant to models_list.go with documentation
  2. Update provider detection in internal/bedrockclient/bedrockclient.go if needed
  3. Add provider-specific implementation in internal/bedrockclient/provider_*.go
  4. Update tests in bedrockllm_test.go to include new model
  5. For caching support, add pattern to supportsCaching() method

When updating API:

  1. Converse API changes go to internal/bedrockclient/bedrockclient_converse.go
  2. Legacy API changes go to internal/bedrockclient/provider_*.go
  3. Message processing changes go to bedrockllm.go (processMessages, processMessagesWithCaching)
  4. Always maintain backward compatibility
  5. Add integration tests with httprr recording

Testing

Tests use httprr for HTTP recording/replay:

  • Integration tests: bedrockllm_test.go (requires AWS credentials)
  • Unit tests: bedrockllm_unit_test.go (no credentials needed)
  • Tool calling: bedrock_tool_integration_test.go

Recording new HTTP interactions:

HTTPRR_RECORD=. go test -v -run TestName ./llms/bedrock/

Debug HTTP interactions:

HTTPRR_RECORD=. HTTPRR_DEBUG=true go test -v -run TestName ./llms/bedrock/

Index

Constants

View Source
const (
	// Amazon Nova 2 Lite is an advanced multimodal model geared towards adaptive reasoning, efficient thinking,
	// customization and agentic workflows. It intelligently balances performance and efficiency by dynamically
	// adjusting reasoning depth based on task complexity.
	//
	// Max tokens: 1M
	// Languages: 200+ languages (optimized for English, German, Spanish, French, Italian, Japanese, Korean, Arabic, Simplified Chinese, Russian, Hindi, Portuguese, Dutch, Turkish, and Hebrew).
	ModelAmazonNova2LiteV1 = "us.amazon.nova-2-lite-v1:0"

	// Amazon Nova Pro is a multimodal understanding foundation model. It is multilingual and can reason
	// over text, images and videos. It supports agents, chat optimization, code generation, complex
	// reasoning analysis, conversation, math, multilingual support, question answering, RAG, text
	// generation, text summarization, translation, and video-to-text.
	//
	// Max tokens: 300k
	// Languages: 200+ languages.
	ModelAmazonNovaProV1 = "us.amazon.nova-pro-v1:0"

	// Amazon Nova Lite is a multimodal understanding foundation model. It is multilingual and can reason
	// over text, images and videos. It supports agents, chat optimization, conversation, math, multilingual
	// support, question answering, RAG, text generation, text summarization, translation, and video-to-text.
	//
	// Max tokens: 300k
	// Languages: 200+ languages.
	ModelAmazonNovaLiteV1 = "us.amazon.nova-lite-v1:0"

	// Amazon Nova Micro is a text-to-text understanding foundation model. It is multilingual and can reason
	// over text. It supports agents, chat optimization, conversation, math, multilingual support, question
	// answering, RAG, text generation, text summarization, and translation.
	//
	// Max tokens: 128k
	// Languages: 200+ languages.
	ModelAmazonNovaMicroV1 = "us.amazon.nova-micro-v1:0"

	// Claude Fable 5 is Anthropic's most capable widely released model, built for the most demanding
	// reasoning and long-horizon agentic work. Thinking is always on (adaptive); the raw chain of thought
	// is never returned. On Bedrock it requires opting into the provider data-share retention mode, and
	// temperature must be 1.0 or unset.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeFable5 = "us.anthropic.claude-fable-5"

	// Claude Opus 5 is Anthropic's model for complex agentic coding and enterprise work, succeeding
	// Opus 4.8. Adaptive thinking is on by default (a breaking change from Opus 4.8, which defaulted
	// off) but, unlike Fable 5/Mythos 5, it still accepts an explicit disable. Supports adaptive
	// thinking only: budget thinking and sampling params are rejected.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeOpus5 = "us.anthropic.claude-opus-5"

	// Claude Opus 4.8 is Anthropic's most capable Opus-tier model — highly autonomous, state-of-the-art
	// on long-horizon agentic work, knowledge work, and memory, with clearer and warmer writing.
	// Supports adaptive thinking only: budget thinking and sampling params are rejected.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeOpus48 = "us.anthropic.claude-opus-4-8"

	// Claude Opus 4.7 is a highly autonomous previous-generation Opus, strong on long-horizon agentic
	// work, knowledge work, vision, and memory. It introduces the xhigh effort level and high-resolution
	// vision. Supports adaptive thinking only: budget thinking and sampling params are rejected.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeOpus47 = "us.anthropic.claude-opus-4-7"

	// Claude Sonnet 5 is Anthropic's most capable Sonnet, built for coding, agents, and professional
	// work at scale with near-Opus intelligence at Sonnet cost. Adaptive thinking is on by default;
	// budget thinking and non-default sampling params are rejected. On Bedrock, adaptive thinking is
	// always on and cannot be disabled, unlike the Anthropic API where it accepts an explicit disable.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeSonnet5 = "us.anthropic.claude-sonnet-5"

	// Claude Opus 4.6 is the world's best model for coding, enterprise agents, and professional work.
	// It excels at agentic workflows, orchestrating complex tasks across dozens of tools with industry-leading
	// reliability. It handles the full lifecycle from architecture to deployment, delivers the deepest reasoning
	// for security workflows, and is Anthropic's most capable model for financial workflows and computer use.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeOpus46 = "us.anthropic.claude-opus-4-6-v1"

	// Claude Sonnet 4.6 delivers frontier intelligence at scale—built for coding, agents, and enterprise workflows.
	// It excels at complex, multi-step tasks requiring sustained reasoning and adaptive decision-making, handles
	// iterative development work with complex codebases, and brings professional-grade analysis with memory to
	// maintain context across files. Step-change improvement in creating spreadsheets, slides, and docs.
	//
	// Max tokens: 1M
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeSonnet46 = "us.anthropic.claude-sonnet-4-6"

	// Claude Opus 4.5 is the next generation of Anthropic's most intelligent model, an industry leader
	// across coding, agents, computer use, and enterprise workflows. It can confidently deliver multi-day
	// software development projects in hours, working independently with technical depth.
	//
	// Max tokens: 200k
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeOpus45 = "us.anthropic.claude-opus-4-5-20251101-v1:0"

	// Claude Haiku 4.5 delivers near-frontier performance for a wide range of use cases, and stands out
	// as one of the best coding and agent models—with the right speed and cost to power free products
	// and high-volume user experiences.
	//
	// Max tokens: 200k
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeHaiku45 = "us.anthropic.claude-haiku-4-5-20251001-v1:0"

	// Claude Sonnet 4.5 is Anthropic's most powerful model for powering real-world agents, with industry-leading
	// capabilities around coding and computer use. It is the ideal balance of performance and practicality
	// for most internal and external use cases.
	//
	// Max tokens: 200k
	// Languages: English, French, Modern Standard Arabic, Mandarin Chinese, Hindi, Spanish, Portuguese, Korean, Japanese, German, Russian, Polish, and other languages.
	ModelAnthropicClaudeSonnet45 = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"

	// Llama 4 Maverick offers unparalleled, industry-leading performance in image and text understanding
	// with support for 12 languages, enabling the creation of sophisticated AI applications that bridge
	// language barriers. As the product workhorse model for general assistant and chat use cases,
	// it's great for precise image understanding and creative writing.
	//
	// Max tokens: 1M
	// Languages: English, French, German, Hindi, Italian, Portuguese, Spanish, Thai, Arabic, Indonesian, Tagalog, Vietnamese.
	ModelMetaLlama4MaverickInstructV1 = "us.meta.llama4-maverick-17b-instruct-v1:0"

	// Llama 4 Scout is a general purpose model with 17 billion active parameters, 16 experts, and 109 billion
	// total parameters that delivers state-of-the-art performance for its class. Scout dramatically increases
	// the supported context length to an industry leading 10 million tokens, opening up possibilities for
	// multi-document summarization, parsing extensive user activity, and reasoning over vast codebases.
	//
	// Max tokens: 3.5M
	// Languages: English, French, German, Hindi, Italian, Portuguese, Spanish, Thai, Arabic, Indonesian, Tagalog, Vietnamese.
	ModelMetaLlama4ScoutInstructV1 = "us.meta.llama4-scout-17b-instruct-v1:0"

	// Llama 3.3 70B offers on par performance with the 405B model at a lower cost.
	// With tool use, code generation, advanced reasoning and decision making, and steerability.
	// We recommend upgrading to this model as soon as possible for optimal performance.
	//
	// Max tokens: 128k
	// Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
	ModelMetaLlama3370bInstructV1 = "us.meta.llama3-3-70b-instruct-v1:0"

	// Llama 3.1 70B Instruct is an update to Meta Llama 3 70B Instruct that includes an expanded 128K context length,
	// multilinguality and improved reasoning capabilities. It's optimized for multilingual dialogue use cases
	// and outperforms many available open source chat models on common industry benchmarks.
	//
	// Max tokens: 128k
	// Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
	ModelMetaLlama3170bInstructV1 = "us.meta.llama3-1-70b-instruct-v1:0"

	// Llama 3.1 8B Instruct is an update to Meta Llama 3 8B Instruct that includes an expanded 128K context length,
	// multilinguality and improved reasoning capabilities. It's optimized for multilingual dialogue use cases
	// and outperforms many available open source chat models on common industry benchmarks.
	//
	// Max tokens: 128k
	// Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
	ModelMetaLlama318bInstructV1 = "meta.llama3-1-8b-instruct-v1:0"

	// Meta Llama 3 70B Instruct is an accessible, open large language model designed for developers,
	// researchers, and businesses to build, experiment, and responsibly scale their generative AI ideas.
	// Ideal for content creation, conversational AI, language understanding, R&D, and Enterprise applications.
	//
	// Max tokens: 8k
	// Languages: English.
	ModelMetaLlama370bInstructV1 = "meta.llama3-70b-instruct-v1:0"

	// Meta Llama 3 8B Instruct is an accessible, open large language model designed for developers,
	// researchers, and businesses to build, experiment, and responsibly scale their generative AI ideas.
	// Ideal for limited computational power and resources, edge devices, and faster training times.
	//
	// Max tokens: 8k
	// Languages: English.
	ModelMetaLlama38bInstructV1 = "meta.llama3-8b-instruct-v1:0"

	// DeepSeek-V3.2 harmonizes high computational efficiency with superior reasoning and agent performance.
	// It builds on DeepSeek Sparse Attention for long-context efficiency, a scalable reinforcement learning framework,
	// and a large-scale agentic task synthesis pipeline. This model excels at long-context reasoning and agentic tasks,
	// efficiently handling extended inputs while maintaining strong accuracy. Its sparse attention design enables it to
	// process complex, multi-step workflows without excessive compute costs. Targets long-context reasoning,
	// tool-using agents, and efficient deployment in production environments.
	//
	// Max tokens: 164k
	// Languages: English, Chinese.
	ModelDeepSeekV32 = "deepseek.v3.2"

	// DeepSeek-R1 provides customers a state-of-the-art reasoning model, optimized for general reasoning tasks,
	// math, science, and code generation. This model is created by DeepSeek and developed through a combination
	// of cold-start data and reinforcement learning. DeepSeek-R1 is a text-only model supporting English and Chinese.
	//
	// Max tokens: 128k
	// Languages: English, Chinese.
	ModelDeepSeekR1V1 = "us.deepseek.r1-v1:0"

	// OpenAI GPT-OSS-120B delivers performance comparable to and surpassing leading alternatives, particularly
	// in coding, scientific analysis, and mathematical reasoning tasks. It excels in intelligent automation,
	// software development, complex problem-solving, and scientific research applications.
	//
	// Max tokens: 128k
	// Languages: English.
	ModelOpenAIGptOss120BV1 = "openai.gpt-oss-120b-1:0"

	// OpenAI GPT-OSS-20B delivers performance comparable to and surpassing leading alternatives, particularly
	// in coding, scientific analysis, and mathematical reasoning tasks. It excels in intelligent automation,
	// software development, complex problem-solving, and scientific research applications.
	//
	// Max tokens: 128k
	// Languages: English.
	ModelOpenAIGptOss20BV1 = "openai.gpt-oss-20b-1:0"

	// Qwen3 Next 80B A3B turns cutting-edge MoE and hybrid attention into a practical, ultra-long-context assistant
	// that scales from everyday chat to million-token workflows. It delivers flagship-level reasoning, coding,
	// and agent performance with only 3B active parameters per token. Ideal for long-context summarization,
	// code generation/refactoring, enterprise knowledge QA, and stable agentic workflows with tools.
	//
	// Max tokens: 256k
	// Languages: English, Chinese.
	ModelQwen3Next80BA3B = "qwen.qwen3-next-80b-a3b"

	// Qwen3 VL 235B A22B is a frontier vision-language model that sees, reads, and reasons across images, documents,
	// and long videos at massive scale. Its 235B-parameter MoE architecture (≈22B active) delivers state-of-the-art
	// multimodal understanding, OCR, and spatial reasoning over contexts reaching hundreds of thousands of tokens.
	// Ideal for document intelligence (OCR + layout), multimodal RAG, visual QA, and UI/scene understanding.
	//
	// Max tokens: 256k
	// Languages: English, Chinese.
	ModelQwen3VL235BA22B = "qwen.qwen3-vl-235b-a22b"

	// Qwen3 32B is a balanced dense model that offers strong reasoning and general-purpose performance with
	// straightforward deployment on standard infrastructure. Despite its smaller size compared to frontier-scale
	// models, Qwen3-32B delivers performance that surpasses many larger models, and proves highly versatile across
	// reasoning, coding, and research use cases. Its balance of capability, cost efficiency, and operational
	// simplicity has made it one of the most practical and widely deployable models in the Qwen3 family.
	//
	// Max tokens: 16384
	// Languages: English, Chinese.
	ModelQwen332BV1 = "qwen.qwen3-32b-v1:0"

	// Qwen3-Coder-30B-A3B-Instruct delivers strong coding and reasoning performance in a compact MoE design, making
	// it one of the most widely adopted models in the Qwen3-Coder series. It has become a favorite among developers
	// and enterprises seeking a practical balance between cost and capability. The model excels at "vibe coding,"
	// natural-language-first programming, debugging, SQL generation, and other development workflows, while being
	// lightweight enough to run on single high-memory GPUs or small clusters.
	//
	// Max tokens: 262144
	// Languages: English, Chinese.
	ModelQwen3Coder30BA3BV1 = "qwen.qwen3-coder-30b-a3b-v1:0"

	// Qwen3-Coder-Next is an open-weight language model built specifically for coding, with strong performance
	// on large-scale software engineering and agentic coding benchmarks. It uses a hybrid Mixture-of-Experts
	// architecture to offer high capability at relatively modest active parameter counts, improving efficiency
	// for real-world deployments. Optimized for tool use and function calling, making it suitable as the core
	// of coding agents that interact with shells, editors, issue trackers, and other developer tools.
	//
	// Max tokens: 256k
	// Languages: English, Chinese.
	ModelQwen3CoderNext = "qwen.qwen3-coder-next"

	// Mistral Large 3 is Mistral's most advanced open-weight multimodal model, combining a granular
	// Mixture-of-Experts architecture (673B total parameters with 39B active, plus a 2.5B vision encoder)
	// and a 256k context window to deliver state-of-the-art reliability, long-context reasoning, and
	// agentic performance for production assistants, RAG systems, scientific workloads, and complex enterprise applications.
	//
	// Max tokens: 256k
	// Languages: English, French, Spanish, German, Russian, Chinese, Japanese, Italian, Portuguese, Dutch, Polish, Vietnamese, Indonesian, Czech, Turkish, Farsi, Greek, Swedish, Arabic, Hungarian, Romanian, Finnish, Danish, Norwegian, Hebrew, Catalan, Hindi, Korean, Bengali, Tamil, Serbian, Urdu, Nepali, Marathi, Croatian, Telugu, Khmer, Tagalog, Gujarati, Malay, Kannada, Punjabi, Lao, Breton.
	ModelMistralLarge3 = "mistral.mistral-large-3-675b-instruct"

	// Devstral 2 123B is Mistral's 123-billion parameter (FP8) agentic model purpose-built for
	// software engineering: autonomous coding workflows, multi-file edits, and native tool-calling
	// to explore repositories and orchestrate complex engineering tasks. Scores 72.2% on SWE-bench
	// Verified and 61.3% on SWE-bench Multilingual.
	//
	// Max tokens: 256k
	// Languages: English (primary), French, Spanish, German, Italian, Portuguese, Chinese, Japanese, Korean, and 20+ additional languages.
	ModelMistralDevstral2123B = "mistral.devstral-2-123b"

	// Magistral Small 2509 is Mistral's small-sized dense model optimized for fast, cost-efficient instruction
	// following, reasoning, and coding, designed as a production-friendly "small but capable" assistant.
	// It brings "big model" quality to a smaller form factor with multimodal support for vision and text.
	//
	// Max tokens: 128k
	// Languages: English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi (24 languages total).
	ModelMistralMagistralSmall2509 = "mistral.magistral-small-2509"

	// Mistral Large (24.02) is the most advanced Mistral AI Large Language model capable of handling any language task
	// including complex multilingual reasoning, text understanding, transformation, and code generation.
	//
	// Max tokens: 32k
	// Languages: English, French, German, Spanish, Chinese, Japanese, and multiple other languages.
	ModelMistralLarge2402V1 = "mistral.mistral-large-2402-v1:0"

	// Kimi K2.5 brings together strong vision, language, and code capabilities in a single natively multimodal
	// architecture. It handles complex tasks that mix images and text—such as generating code from UI mockups
	// or analyzing visual documents—with high accuracy. The model's "thinking" mode enables deep, deliberate reasoning,
	// while "instant" mode provides fast responses for interactive use. Its built-in support for tool use and agent
	// orchestration makes it highly effective for building sophisticated multimodal assistants.
	//
	// Max tokens: 256k
	// Languages: English, Chinese.
	ModelMoonshotKimiK25 = "moonshotai.kimi-k2.5"

	// Kimi K2 Thinking is Moonshot AI's flagship "thinking agent" model, designed for deep, tool-augmented reasoning.
	// Its 1T-parameter MoE architecture (32B active) powers state-of-the-art performance on long-horizon tasks like
	// HLE and BrowseComp. Native INT4 quantization and a 256K context window enable serious research- and agent-style
	// workloads with practical hardware. Ideal for long-horizon planning with tools, complex coding and debugging,
	// research agents over large corpora, and workflows needing 200-300-step stable tool orchestration.
	//
	// Max tokens: 256k
	// Languages: Multilingual (including Chinese and English).
	ModelMoonshotKimiK2Thinking = "moonshot.kimi-k2-thinking"

	// GLM-4.7 is a general-purpose language model in the GLM family with a focus on generating clean, modern
	// front-end code and web interfaces. It can turn natural-language descriptions into structured HTML, CSS,
	// and JavaScript while also supporting standard text and reasoning tasks. The model is positioned for developers
	// who want high-quality UI outputs alongside general conversational and coding abilities. It remains compatible
	// with typical LLM use cases such as question answering, summarization, and dialogue.
	//
	// Max tokens: 203k
	// Languages: English, Chinese.
	ModelGLM47 = "zai.glm-4.7"

	// GLM-4.7-Flash is a lightweight variant of GLM-4.7, using a mixture-of-experts architecture to reduce
	// resource requirements while maintaining strong output quality. It is designed for scenarios where low latency
	// and cost efficiency are important, such as interactive assistants or high-traffic services. The model retains
	// the core text and code generation capabilities of GLM-4.7 in a smaller active-parameter footprint.
	// A practical choice when deployment constraints limit the use of larger models.
	//
	// Max tokens: 203k
	// Languages: English, Chinese.
	ModelGLM47Flash = "zai.glm-4.7-flash"

	// GLM-5 is Z.ai's frontier reasoning and agentic model, a significant step up from the GLM-4.x line
	// on complex reasoning, coding, and multi-step agent workflows. It targets production agent systems
	// that need strong tool use and long-context planning.
	//
	// Max tokens: 200k
	// Languages: English, Chinese.
	ModelGLM5 = "zai.glm-5"

	// MiniMax M2.5 is an agent-native frontier model trained to reason efficiently, decompose tasks
	// optimally, and complete complex workflows under real-world time and cost constraints. Well
	// suited for production agents handling full-stack software projects, research and analysis
	// workflows, long-horizon planning, and multi-tool orchestration.
	//
	// Max tokens: 196k
	// Languages: English, Chinese.
	ModelMiniMaxM25 = "minimax.minimax-m2.5"

	// MiniMax M2.1 is an open-weight model focused on coding, tool use, and long-horizon task
	// planning, evaluated on practical front-end, backend, and workflow-automation benchmarks.
	// Intended as a general-purpose backbone for agent-based applications with improved reasoning,
	// coding, and instruction following over M2.
	//
	// Max tokens: 196k
	// Languages: English, Chinese.
	ModelMiniMaxM21 = "minimax.minimax-m2.1"

	// MiniMax M2 is a MoE model that blends frontier-level intelligence with highly efficient
	// active parameters, engineered for AI agents with strong reasoning, coding, and multilingual
	// performance at competitive cost. Suited for general-purpose chat/coding, tool-using agents,
	// multilingual assistants, and high-throughput inference.
	//
	// Max tokens: 400k
	// Languages: English, Chinese.
	ModelMiniMaxM2 = "minimax.minimax-m2"

	// NVIDIA Nemotron 3 Super 120B is an open hybrid mixture-of-experts model (about 12B active
	// parameters) built for reasoning, coding, and agentic tasks with strong cost efficiency.
	//
	// Max tokens: 256k
	// Languages: English.
	ModelNvidiaNemotronSuper3120B = "nvidia.nemotron-super-3-120b"
)

Variables

This section is empty.

Functions

func EphemeralCache

func EphemeralCache() *llms.CacheControl

EphemeralCache creates a standard ephemeral cache control for Bedrock with 5-minute duration.

func EphemeralCacheOneHour

func EphemeralCacheOneHour() *llms.CacheControl

EphemeralCacheOneHour creates a 1-hour ephemeral cache control for Bedrock. Supported by Claude Opus 4.5, Haiku 4.5, and Sonnet 4.5.

func MapError

func MapError(err error) error

MapError maps AWS Bedrock-specific errors to standardized error codes.

Types

type CachedContent

type CachedContent struct {
	llms.ContentPart
	CacheControl *llms.CacheControl `json:"cache_control,omitempty"`
}

CachedContent represents content with caching instructions for Bedrock. This wraps any ContentPart and adds cache control metadata.

Note: For most use cases, prefer using bedrock.WithAutomaticCaching() option which automatically applies caching to supported Anthropic models (Claude 4.x). This manual wrapper is only needed for fine-grained cache control.

Automatic caching is supported in both Legacy and Converse APIs.

func WithCacheControl

func WithCacheControl(content llms.ContentPart, control *llms.CacheControl) CachedContent

WithCacheControl wraps content with cache control instructions for Bedrock. This allows explicit control over what content should be cached.

Recommended: Use bedrock.WithAutomaticCaching() option instead for transparent caching.

Manual usage (when fine-grained control is needed):

bedrock.WithCacheControl(
    llms.TextPart("long context..."),
    bedrock.EphemeralCache(),
)

Supported models: Claude Opus 4, Sonnet 4, Haiku 4 and their variants.

type LLM

type LLM struct {
	CallbacksHandler callbacks.Handler
	// contains filtered or unexported fields
}

LLM is a Bedrock LLM implementation.

func New

func New(opts ...Option) (*LLM, error)

New creates a new Bedrock LLM implementation.

func NewWithContext

func NewWithContext(ctx context.Context, opts ...Option) (*LLM, error)

NewWithContext creates a new Bedrock LLM implementation with context.

func (*LLM) Call

func (l *LLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)

Call implements llms.Model.

func (*LLM) GenerateContent

func (l *LLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (resp *llms.ContentResponse, err error)

GenerateContent implements llms.Model.

type Option

type Option func(*options)

Option is an option for the Bedrock LLM.

func WithAutomaticCaching

func WithAutomaticCaching() Option

WithAutomaticCaching enables automatic prompt caching for supported Anthropic models.

When enabled, caching is automatically applied for models matching these patterns: - claude-opus-4 (includes 4.6, 4.5, 4.1, 4.0) - claude-sonnet-4 (includes 4.6, 4.5, 4.0) - claude-haiku-4 (includes 4.5)

The caching strategy automatically: - Adds cache points to system prompts - Adds cache points to conversation history (last message before new user input) - Uses ephemeral 5-minute TTL by default

Benefits: - 90% cost reduction on cached input tokens - No manual cache control wrapper needed on client side - Transparent caching without modifying message chains

Note: Automatic caching works with both Legacy and Converse APIs.

func WithCallback

func WithCallback(callbackHandler callbacks.Handler) Option

WithCallback allows setting a custom Callback Handler.

func WithClient

func WithClient(client *bedrockruntime.Client) Option

WithClient allows setting a custom bedrockruntime.Client.

You may use this to pass a custom bedrockruntime.Client with custom configuration options such as setting custom credentials, region, endpoint, etc.

By default, a new client will be created using the default credentials chain.

func WithConverseAPI

func WithConverseAPI() Option

WithConverseAPI enables the use of the unified Bedrock Converse API instead of the model-specific legacy implementations.

The Converse API provides: - Unified interface for all supported Bedrock models - Built-in tool calling support - Streaming responses with ConverseStream - Reasoning content support for Claude 3.7+ and Nova models - Multimodal input support (text, images, documents) - Better error handling and response consistency - Prompt caching support via cachePoint (requires AWS SDK types)

Supported models: All Anthropic Claude, Amazon Nova, Meta Llama, Cohere Command, and AI21 Jamba models available through Bedrock.

Note: This is the recommended approach for new applications.

Prompt Caching: - Legacy API (InvokeModel) supports Anthropic's cache_control format - Converse API supports cachePoint via SystemContentBlockMemberCachePoint - Cache metrics are returned in response.Usage (CacheReadInputTokens, CacheWriteInputTokens) - Requires minimum tokens per checkpoint (1024 for Sonnet 4.5, 4096 for Haiku 4.5) - Supports 5m and 1h TTL for Claude 4.x models

func WithModel

func WithModel(modelID string) Option

WithModel allows setting a custom modelId.

If not set, the default model is used i.e. "amazon.titan-text-lite-v1".

func WithModelProvider

func WithModelProvider(modelProvider string) Option

WithModelProvider allows setting a custom model provider.

If not set, the default model provider is used i.e. "anthropic".

Directories

Path Synopsis
internal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL