Documentation
¶
Index ¶
- Constants
- func ArtifactKeys(platformKey, vendor string) []string
- func BackendForVendor(vendor string) string
- func DecodePublicKey(encoded string) (ed25519.PublicKey, error)
- func DefaultRuntimePublicKey() (ed25519.PublicKey, error)
- type ApprovedArgs
- type ChatMessage
- type CompletionRequest
- type CompletionResponse
- type CompletionUsage
- type InstalledRuntime
- type LaunchConfig
- type Manager
- type ManifestSignature
- type ReleaseManifest
- type RunningRuntime
- type RuntimeArtifact
- type SupportArtifact
Constants ¶
const ( VendorNvidia = "nvidia" VendorAMD = "amd" VendorUnknown = "unknown" )
GPU vendor identifiers used to pick a vendor-specific runtime build. These mirror hardware.Vendor* but are duplicated here so the runtime package does not depend on hardware detection.
const ( BackendCUDA = "cuda" BackendVulkan = "vulkan" )
llama.cpp compute backends. Manifest artifact keys are named by backend, not by vendor, because the backend is what actually differs between builds and several vendors can share one (Vulkan runs on AMD and Intel alike).
const DefaultRuntimePublicKeyBase64 = "/Bgu4fQntQGB6q3j18Z+du1L1/yOzU2/wNmSAQWVCMo="
Variables ¶
This section is empty.
Functions ¶
func ArtifactKeys ¶
ArtifactKeys lists the manifest keys to try, most specific first. A backend-specific build is preferred when the manifest carries one; the bare platform key stays as the fallback so nodes that predate backend keys, and manifests that never gained them, keep working unchanged.
func BackendForVendor ¶
BackendForVendor maps a detected GPU vendor onto the runtime build it needs.
An undetected vendor resolves to Vulkan rather than to the bare platform artifact, because the failure modes are not symmetric: the bare Windows artifact is the CUDA build, which cannot run at all on a Radeon, while the Vulkan build runs on NVIDIA too, only slower. When detection fails the safe guess is the one that degrades instead of breaking.
func DefaultRuntimePublicKey ¶
Types ¶
type ApprovedArgs ¶
type ChatMessage ¶
type CompletionRequest ¶
type CompletionRequest struct {
Model string `json:"model"`
Messages []ChatMessage `json:"messages"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
Stream bool `json:"stream"`
}
type CompletionResponse ¶
type CompletionResponse struct {
Choices []struct {
Message ChatMessage `json:"message"`
FinishReason string `json:"finish_reason"`
} `json:"choices"`
Usage CompletionUsage `json:"usage"`
}
func ChatCompletion ¶
func ChatCompletion(ctx context.Context, baseURL string, request CompletionRequest) (CompletionResponse, error)
type CompletionUsage ¶
type InstalledRuntime ¶
type InstalledRuntime struct {
Engine string `json:"engine"`
Version string `json:"version"`
BuildID string `json:"build_id"`
PlatformKey string `json:"platform_key"`
Path string `json:"path"`
ExecutablePath string `json:"executable_path"`
SHA256 string `json:"sha256"`
InstalledAt string `json:"installed_at"`
}
type LaunchConfig ¶
type Manager ¶
type Manager struct {
Root string
PublicKey ed25519.PublicKey
PlatformKey string
// GPUVendor picks a vendor-specific runtime build when the manifest has
// one. Empty or "unknown" falls back to the bare platform artifact.
GPUVendor string
HTTPClient *http.Client
}
func (Manager) Ensure ¶
func (m Manager) Ensure(ctx context.Context, manifest ReleaseManifest) (InstalledRuntime, error)
func (Manager) Rollback ¶
func (m Manager) Rollback() (InstalledRuntime, error)
type ManifestSignature ¶
type ReleaseManifest ¶
type ReleaseManifest struct {
SchemaVersion int `json:"schema_version"`
Engine string `json:"engine"`
Version string `json:"version"`
BuildID string `json:"build_id"`
Artifacts map[string]RuntimeArtifact `json:"artifacts"`
Signature ManifestSignature `json:"signature"`
}
func LoadReleaseManifest ¶
func LoadReleaseManifest(path string) (ReleaseManifest, error)
func ParseReleaseManifest ¶
func ParseReleaseManifest(data []byte, source string) (ReleaseManifest, error)
ParseReleaseManifest decodes a runtime release manifest from bytes; source is used only for error messages.
func (ReleaseManifest) SelectArtifact ¶
func (m ReleaseManifest) SelectArtifact(platformKey, vendor string) (RuntimeArtifact, string, error)
SelectArtifact resolves the artifact for a platform and GPU vendor, returning the artifact and the key it was found under.
func (ReleaseManifest) UnsignedBytes ¶
func (m ReleaseManifest) UnsignedBytes() ([]byte, error)
func (ReleaseManifest) VerifySignature ¶
func (m ReleaseManifest) VerifySignature(publicKey ed25519.PublicKey) error
type RunningRuntime ¶
func StartLlamaServer ¶
func StartLlamaServer(ctx context.Context, cfg LaunchConfig) (*RunningRuntime, error)