Documentation
¶
Overview ¶
Package provisioner rents on-demand GPU instances running llama.cpp and exposes them as a llamacpp provider endpoint (issue #939). RunPod is the first driver; the surface is plain CRUD on instances.
Index ¶
- Constants
- type GPUType
- type Pod
- type ProvisionRequest
- type RunPod
- func (r *RunPod) Destroy(ctx context.Context, id string) error
- func (r *RunPod) GPUTypes(ctx context.Context) ([]GPUType, error)
- func (r *RunPod) List(ctx context.Context) ([]Pod, error)
- func (r *RunPod) Provision(ctx context.Context, req ProvisionRequest) (Pod, error)
- func (r *RunPod) Status(ctx context.Context, id string) (Pod, error)
- func (r *RunPod) WaitReady(ctx context.Context, id string, port int, report func(string)) (Pod, error)
Constants ¶
const PodNamePrefix = "infer-"
PodNamePrefix tags every pod created by infer so List can find them.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type GPUType ¶
type GPUType struct {
ID string `json:"id"`
DisplayName string `json:"displayName"`
MemoryInGb int `json:"memoryInGb"`
CommunityPrice float64 `json:"communityPrice"`
SecurePrice float64 `json:"securePrice"`
}
GPUType is a rentable GPU with live pricing.
type Pod ¶
type Pod struct {
ID string `json:"id"`
Name string `json:"name"`
Image string `json:"image"`
DesiredStatus string `json:"desiredStatus"`
CostPerHr float64 `json:"costPerHr"`
LastStartedAt string `json:"lastStartedAt"`
}
Pod is the subset of RunPod's Pod object the CLI cares about.
type ProvisionRequest ¶
type ProvisionRequest struct {
Name string
GPUTypeID string
Image string
StartCmd []string
CloudType string // COMMUNITY or SECURE
DiskGB int
Env map[string]string
Ports []string
}
ProvisionRequest describes the pod to create.
type RunPod ¶
type RunPod struct {
APIKey string
RestURL string // default https://rest.runpod.io/v1
GraphQLURL string // default https://api.runpod.io/graphql
ProxyBase string // overrides Pod.ProxyURL in WaitReady; tests only
Client *http.Client
// contains filtered or unexported fields
}
RunPod is the RunPod driver. Base URLs are fields so tests can point at httptest.
func (*RunPod) GPUTypes ¶
GPUTypes returns rentable GPU types with live pricing, cheapest first. This is the one non-REST call (RunPod exposes pricing only via GraphQL).
func (*RunPod) WaitReady ¶
func (r *RunPod) WaitReady(ctx context.Context, id string, port int, report func(string)) (Pod, error)
WaitReady polls the pod, then the llama.cpp /health endpoint through the proxy until the model answers. /health is unauthenticated in llama.cpp, so this needs no per-session token and works for `gpu status --wait` too.
Source Files