llm

package
v1.7.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 7, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

SPDX-License-Identifier: MIT Purpose: NVIDIA NIM-specific helpers. NIM exposes an OpenAI-compatible chat/completions endpoint at https://integrate.api.nvidia.com/v1 and serves Anthropic / Llama / Mistral / etc. models under "vendor/name" IDs. Friendly aliases like "haiku" or "sonnet" resolve to the full ID.

SPDX-License-Identifier: MIT Purpose: generic OpenAI-compatible LLM client. Single-shot chat completion request with bearer auth, JSON marshaling, and typed response decoding.

Index

Constants

View Source
const NIMClaudeModel = "anthropic/claude-3-5-sonnet-20241022"
View Source
const NIMDefaultBaseURL = "https://integrate.api.nvidia.com/v1"
View Source
const NIMDefaultModel = "meta/llama-3.1-70b-instruct"
View Source
const NIMHaikuModel = "anthropic/claude-3-5-haiku-20241022"

Variables

View Source
var NIMModelAliases = map[string]string{
	"haiku":     NIMHaikuModel,
	"sonnet":    NIMClaudeModel,
	"llama-70b": NIMDefaultModel,
	"llama-8b":  "meta/llama-3.1-8b-instruct",
}

Functions

func ResolveModel

func ResolveModel(name string) string

Types

type ChatRequest

type ChatRequest struct {
	Model       string    `json:"model"`
	Messages    []Message `json:"messages"`
	MaxTokens   int       `json:"max_tokens,omitempty"`
	Temperature float64   `json:"temperature,omitempty"`
	Stream      bool      `json:"stream,omitempty"`
}

type ChatResponse

type ChatResponse struct {
	ID      string `json:"id"`
	Object  string `json:"object"`
	Created int64  `json:"created"`
	Model   string `json:"model"`
	Choices []struct {
		Index        int     `json:"index"`
		Message      Message `json:"message"`
		FinishReason string  `json:"finish_reason"`
	} `json:"choices"`
	Usage struct {
		PromptTokens     int `json:"prompt_tokens"`
		CompletionTokens int `json:"completion_tokens"`
		TotalTokens      int `json:"total_tokens"`
	} `json:"usage"`
}

func (*ChatResponse) ExtractText

func (r *ChatResponse) ExtractText() string

type Client

type Client struct {
	BaseURL string
	APIKey  string
	HTTP    *http.Client
}

func NewClient

func NewClient(baseURL, apiKey string) *Client

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL