worker

package
v0.8.10 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 10, 2026 License: MIT Imports: 46 Imported by: 0

Documentation

Overview

Package `worker` hosts the main AI worker logic for managing or using runner containers for processing inference requests on the Livepeer AI subnet. The package allows interacting with the AI runner containers, and it includes:

- Golang API Bindings (./runner.gen.go):

Generated from the AI runner's OpenAPI spec. To re-generate them run: `make ai_worker_codegen`

- Worker (./worker.go):

Listens for inference requests from the Livepeer AI subnet and routes them to the AI runner.

- Docker Manager (./docker.go):

Manages AI runner containers. For a state diagram showing the lifecycle of a container, see the /doc/worker.md file.

Package worker's module provides multipart form data utilities.

This module extends oapi-codegen Go bindings with multipart writers, enabling request parameter encoding for inter-node communication (gateway, orchestrator, runner).

Package worker provides primitives to interact with the openapi HTTP API.

Code generated by github.com/deepmap/oapi-codegen/v2 version v2.2.0 DO NOT EDIT.

Index

Constants

View Source
const (
	HTTPBearerScopes = "HTTPBearer.Scopes"
)

Variables

This section is empty.

Functions

func GetSwagger

func GetSwagger() (swagger *openapi3.T, err error)

GetSwagger returns the Swagger specification corresponding to the generated code in this file. The external references of Swagger specification are resolved. The logic of resolving external references is tightly connected to "import-mapping" feature. Externally referenced files must be embedded in the corresponding golang packages. Urls can be supported but this task was out of the scope.

func Handler

func Handler(si ServerInterface) http.Handler

Handler creates http.Handler with routing matching OpenAPI spec.

func HandlerFromMux

func HandlerFromMux(si ServerInterface, r chi.Router) http.Handler

HandlerFromMux creates http.Handler with routing matching OpenAPI spec based on the provided mux.

func HandlerFromMuxWithBaseURL

func HandlerFromMuxWithBaseURL(si ServerInterface, r chi.Router, baseURL string) http.Handler

func HandlerWithOptions

func HandlerWithOptions(si ServerInterface, options ChiServerOptions) http.Handler

HandlerWithOptions creates http.Handler with additional options

func LowestVersion added in v0.8.6

func LowestVersion(versions []Version, pipeline string, modelId string) string

LowestVersion returns the lowest version of a given pipeline and model ID from a list of versions.

func NewGenAudioToTextRequestWithBody

func NewGenAudioToTextRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenAudioToTextRequestWithBody generates requests for GenAudioToText with any type of body

func NewGenImageToImageRequestWithBody

func NewGenImageToImageRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenImageToImageRequestWithBody generates requests for GenImageToImage with any type of body

func NewGenImageToTextRequestWithBody

func NewGenImageToTextRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenImageToTextRequestWithBody generates requests for GenImageToText with any type of body

func NewGenImageToVideoRequestWithBody

func NewGenImageToVideoRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenImageToVideoRequestWithBody generates requests for GenImageToVideo with any type of body

func NewGenLLMRequest

func NewGenLLMRequest(server string, body GenLLMJSONRequestBody) (*http.Request, error)

NewGenLLMRequest calls the generic GenLLM builder with application/json body

func NewGenLLMRequestWithBody

func NewGenLLMRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenLLMRequestWithBody generates requests for GenLLM with any type of body

func NewGenLiveVideoToVideoRequest

func NewGenLiveVideoToVideoRequest(server string, body GenLiveVideoToVideoJSONRequestBody) (*http.Request, error)

NewGenLiveVideoToVideoRequest calls the generic GenLiveVideoToVideo builder with application/json body

func NewGenLiveVideoToVideoRequestWithBody

func NewGenLiveVideoToVideoRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenLiveVideoToVideoRequestWithBody generates requests for GenLiveVideoToVideo with any type of body

func NewGenSegmentAnything2RequestWithBody

func NewGenSegmentAnything2RequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenSegmentAnything2RequestWithBody generates requests for GenSegmentAnything2 with any type of body

func NewGenTextToImageRequest

func NewGenTextToImageRequest(server string, body GenTextToImageJSONRequestBody) (*http.Request, error)

NewGenTextToImageRequest calls the generic GenTextToImage builder with application/json body

func NewGenTextToImageRequestWithBody

func NewGenTextToImageRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenTextToImageRequestWithBody generates requests for GenTextToImage with any type of body

func NewGenTextToSpeechRequest

func NewGenTextToSpeechRequest(server string, body GenTextToSpeechJSONRequestBody) (*http.Request, error)

NewGenTextToSpeechRequest calls the generic GenTextToSpeech builder with application/json body

func NewGenTextToSpeechRequestWithBody

func NewGenTextToSpeechRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenTextToSpeechRequestWithBody generates requests for GenTextToSpeech with any type of body

func NewGenUpscaleRequestWithBody

func NewGenUpscaleRequestWithBody(server string, contentType string, body io.Reader) (*http.Request, error)

NewGenUpscaleRequestWithBody generates requests for GenUpscale with any type of body

func NewHardwareInfoRequest

func NewHardwareInfoRequest(server string) (*http.Request, error)

NewHardwareInfoRequest generates requests for HardwareInfo

func NewHardwareStatsRequest

func NewHardwareStatsRequest(server string) (*http.Request, error)

NewHardwareStatsRequest generates requests for HardwareStats

func NewHealthRequest

func NewHealthRequest(server string) (*http.Request, error)

NewHealthRequest generates requests for Health

func NewUpscaleMultipartWriter

func NewUpscaleMultipartWriter(w io.Writer, req GenUpscaleMultipartRequestBody) (*multipart.Writer, error)

func NewVersionRequest added in v0.8.6

func NewVersionRequest(server string) (*http.Request, error)

NewVersionRequest generates requests for Version

func PathToRawSpec

func PathToRawSpec(pathToFile string) map[string]func() ([]byte, error)

Constructs a synthetic filesystem for resolving external references when loading openapi specifications.

func ReadAudioB64DataUrl

func ReadAudioB64DataUrl(url string, w io.Writer) error

func ReadImageB64DataUrl

func ReadImageB64DataUrl(url string, w io.Writer) error

func RemoveExistingContainers added in v0.8.8

func RemoveExistingContainers(ctx context.Context, client DockerClient, containerCreatorID string) (int, error)

func SaveImageB64DataUrl

func SaveImageB64DataUrl(url, outputPath string) error

Types

type APIError

type APIError struct {
	// Msg The error message.
	Msg string `json:"msg"`
}

APIError API error response model.

type AudioResponse

type AudioResponse struct {
	// Audio The generated audio.
	Audio MediaURL `json:"audio"`
}

AudioResponse Response model for audio generation.

type BodyGenAudioToText

type BodyGenAudioToText struct {
	// Audio Uploaded audio file to be transcribed.
	Audio openapi_types.File `json:"audio"`

	// Metadata Additional job information to be passed to the pipeline.
	Metadata *string `json:"metadata,omitempty"`

	// ModelId Hugging Face model ID used for transcription.
	ModelId *string `json:"model_id,omitempty"`

	// ReturnTimestamps Return timestamps for the transcribed text. Supported values: 'sentence', 'word', or a string boolean ('true' or 'false'). Default is 'true' ('sentence'). 'false' means no timestamps. 'word' means word-based timestamps.
	ReturnTimestamps *string `json:"return_timestamps,omitempty"`
}

BodyGenAudioToText defines model for Body_genAudioToText.

type BodyGenImageToImage

type BodyGenImageToImage struct {
	// GuidanceScale Encourages model to generate images closely linked to the text prompt (higher values may reduce image quality).
	GuidanceScale *float32 `json:"guidance_scale,omitempty"`

	// Image Uploaded image to modify with the pipeline.
	Image openapi_types.File `json:"image"`

	// ImageGuidanceScale Degree to which the generated image is pushed towards the initial image.
	ImageGuidanceScale *float32 `json:"image_guidance_scale,omitempty"`

	// Loras A LoRA (Low-Rank Adaptation) model and its corresponding weight for image generation. Example: { "latent-consistency/lcm-lora-sdxl": 1.0, "nerijs/pixel-art-xl": 1.2}.
	Loras *string `json:"loras,omitempty"`

	// ModelId Hugging Face model ID used for image generation.
	ModelId *string `json:"model_id,omitempty"`

	// NegativePrompt Text prompt(s) to guide what to exclude from image generation. Ignored if guidance_scale < 1.
	NegativePrompt *string `json:"negative_prompt,omitempty"`

	// NumImagesPerPrompt Number of images to generate per prompt.
	NumImagesPerPrompt *int `json:"num_images_per_prompt,omitempty"`

	// NumInferenceSteps Number of denoising steps. More steps usually lead to higher quality images but slower inference. Modulated by strength.
	NumInferenceSteps *int `json:"num_inference_steps,omitempty"`

	// Prompt Text prompt(s) to guide image generation.
	Prompt string `json:"prompt"`

	// SafetyCheck Perform a safety check to estimate if generated images could be offensive or harmful.
	SafetyCheck *bool `json:"safety_check,omitempty"`

	// Seed Seed for random number generation.
	Seed *int `json:"seed,omitempty"`

	// Strength Degree of transformation applied to the reference image (0 to 1).
	Strength *float32 `json:"strength,omitempty"`
}

BodyGenImageToImage defines model for Body_genImageToImage.

type BodyGenImageToText

type BodyGenImageToText struct {
	// Image Uploaded image to transform with the pipeline.
	Image openapi_types.File `json:"image"`

	// ModelId Hugging Face model ID used for transformation.
	ModelId *string `json:"model_id,omitempty"`

	// Prompt Text prompt(s) to guide transformation.
	Prompt *string `json:"prompt,omitempty"`
}

BodyGenImageToText defines model for Body_genImageToText.

type BodyGenImageToVideo

type BodyGenImageToVideo struct {
	// Fps The frames per second of the generated video.
	Fps *int `json:"fps,omitempty"`

	// Height The height in pixels of the generated video.
	Height *int `json:"height,omitempty"`

	// Image Uploaded image to generate a video from.
	Image openapi_types.File `json:"image"`

	// ModelId Hugging Face model ID used for video generation.
	ModelId *string `json:"model_id,omitempty"`

	// MotionBucketId Used for conditioning the amount of motion for the generation. The higher the number the more motion will be in the video.
	MotionBucketId *int `json:"motion_bucket_id,omitempty"`

	// NoiseAugStrength Amount of noise added to the conditioning image. Higher values reduce resemblance to the conditioning image and increase motion.
	NoiseAugStrength *float32 `json:"noise_aug_strength,omitempty"`

	// NumInferenceSteps Number of denoising steps. More steps usually lead to higher quality images but slower inference. Modulated by strength.
	NumInferenceSteps *int `json:"num_inference_steps,omitempty"`

	// SafetyCheck Perform a safety check to estimate if generated images could be offensive or harmful.
	SafetyCheck *bool `json:"safety_check,omitempty"`

	// Seed Seed for random number generation.
	Seed *int `json:"seed,omitempty"`

	// Width The width in pixels of the generated video.
	Width *int `json:"width,omitempty"`
}

BodyGenImageToVideo defines model for Body_genImageToVideo.

type BodyGenSegmentAnything2

type BodyGenSegmentAnything2 struct {
	// Box A length 4 array given as a box prompt to the model, in XYXY format.
	Box *string `json:"box,omitempty"`

	// Image Image to segment.
	Image openapi_types.File `json:"image"`

	// MaskInput A low-resolution mask input to the model, typically from a previous prediction iteration, with the form 1xHxW (H=W=256 for SAM).
	MaskInput *string `json:"mask_input,omitempty"`

	// ModelId Hugging Face model ID used for image generation.
	ModelId *string `json:"model_id,omitempty"`

	// MultimaskOutput If true, the model will return three masks for ambiguous input prompts, often producing better masks than a single prediction.
	MultimaskOutput *bool `json:"multimask_output,omitempty"`

	// NormalizeCoords If true, the point coordinates will be normalized to the range [0,1], with point_coords expected to be with respect to image dimensions.
	NormalizeCoords *bool `json:"normalize_coords,omitempty"`

	// PointCoords Nx2 array of point prompts to the model, where each point is in (X,Y) in pixels.
	PointCoords *string `json:"point_coords,omitempty"`

	// PointLabels Labels for the point prompts, where 1 indicates a foreground point and 0 indicates a background point.
	PointLabels *string `json:"point_labels,omitempty"`

	// ReturnLogits If true, returns un-thresholded mask logits instead of a binary mask.
	ReturnLogits *bool `json:"return_logits,omitempty"`
}

BodyGenSegmentAnything2 defines model for Body_genSegmentAnything2.

type BodyGenUpscale

type BodyGenUpscale struct {
	// Image Uploaded image to modify with the pipeline.
	Image openapi_types.File `json:"image"`

	// ModelId Hugging Face model ID used for upscaled image generation.
	ModelId *string `json:"model_id,omitempty"`

	// NumInferenceSteps Number of denoising steps. More steps usually lead to higher quality images but slower inference. Modulated by strength.
	NumInferenceSteps *int `json:"num_inference_steps,omitempty"`

	// Prompt Text prompt(s) to guide upscaled image generation.
	Prompt string `json:"prompt"`

	// SafetyCheck Perform a safety check to estimate if generated images could be offensive or harmful.
	SafetyCheck *bool `json:"safety_check,omitempty"`

	// Seed Seed for random number generation.
	Seed *int `json:"seed,omitempty"`
}

BodyGenUpscale defines model for Body_genUpscale.

type Capacity added in v0.8.6

type Capacity struct {
	ContainersInUse int
	ContainersIdle  int
}

type ChiServerOptions

type ChiServerOptions struct {
	BaseURL          string
	BaseRouter       chi.Router
	Middlewares      []MiddlewareFunc
	ErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error)
}

type Chunk

type Chunk struct {
	// Text The text of the chunk.
	Text string `json:"text"`

	// Timestamp The timestamp of the chunk.
	Timestamp []interface{} `json:"timestamp"`
}

Chunk A chunk of text with a timestamp.

type Client

type Client struct {
	// The endpoint of the server conforming to this interface, with scheme,
	// https://api.deepmap.com for example. This can contain a path relative
	// to the server, such as https://api.deepmap.com/dev-test, and all the
	// paths in the swagger spec will be appended to the server.
	Server string

	// Doer for performing requests, typically a *http.Client with any
	// customized settings, such as certificate chains.
	Client HttpRequestDoer

	// A list of callbacks for modifying requests which are generated before sending over
	// the network.
	RequestEditors []RequestEditorFn
}

Client which conforms to the OpenAPI3 specification for this service.

func NewClient

func NewClient(server string, opts ...ClientOption) (*Client, error)

Creates a new Client, with reasonable defaults

func (*Client) GenAudioToTextWithBody

func (c *Client) GenAudioToTextWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenImageToImageWithBody

func (c *Client) GenImageToImageWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenImageToTextWithBody

func (c *Client) GenImageToTextWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenImageToVideoWithBody

func (c *Client) GenImageToVideoWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenLLM

func (c *Client) GenLLM(ctx context.Context, body GenLLMJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenLLMWithBody

func (c *Client) GenLLMWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenLiveVideoToVideo

func (c *Client) GenLiveVideoToVideo(ctx context.Context, body GenLiveVideoToVideoJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenLiveVideoToVideoWithBody

func (c *Client) GenLiveVideoToVideoWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenSegmentAnything2WithBody

func (c *Client) GenSegmentAnything2WithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenTextToImage

func (c *Client) GenTextToImage(ctx context.Context, body GenTextToImageJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenTextToImageWithBody

func (c *Client) GenTextToImageWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenTextToSpeech

func (c *Client) GenTextToSpeech(ctx context.Context, body GenTextToSpeechJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenTextToSpeechWithBody

func (c *Client) GenTextToSpeechWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) GenUpscaleWithBody

func (c *Client) GenUpscaleWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) HardwareInfo

func (c *Client) HardwareInfo(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) HardwareStats

func (c *Client) HardwareStats(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) Health

func (c *Client) Health(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

func (*Client) Version added in v0.8.6

func (c *Client) Version(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

type ClientInterface

type ClientInterface interface {
	// GenAudioToTextWithBody request with any body
	GenAudioToTextWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// HardwareInfo request
	HardwareInfo(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

	// HardwareStats request
	HardwareStats(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

	// Health request
	Health(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenImageToImageWithBody request with any body
	GenImageToImageWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenImageToTextWithBody request with any body
	GenImageToTextWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenImageToVideoWithBody request with any body
	GenImageToVideoWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenLiveVideoToVideoWithBody request with any body
	GenLiveVideoToVideoWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	GenLiveVideoToVideo(ctx context.Context, body GenLiveVideoToVideoJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenLLMWithBody request with any body
	GenLLMWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	GenLLM(ctx context.Context, body GenLLMJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenSegmentAnything2WithBody request with any body
	GenSegmentAnything2WithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenTextToImageWithBody request with any body
	GenTextToImageWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	GenTextToImage(ctx context.Context, body GenTextToImageJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenTextToSpeechWithBody request with any body
	GenTextToSpeechWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	GenTextToSpeech(ctx context.Context, body GenTextToSpeechJSONRequestBody, reqEditors ...RequestEditorFn) (*http.Response, error)

	// GenUpscaleWithBody request with any body
	GenUpscaleWithBody(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*http.Response, error)

	// Version request
	Version(ctx context.Context, reqEditors ...RequestEditorFn) (*http.Response, error)
}

The interface specification for the client above.

type ClientOption

type ClientOption func(*Client) error

ClientOption allows setting custom parameters during construction

func WithBaseURL

func WithBaseURL(baseURL string) ClientOption

WithBaseURL overrides the baseURL.

func WithHTTPClient

func WithHTTPClient(doer HttpRequestDoer) ClientOption

WithHTTPClient allows overriding the default Doer, which is automatically created using http.Client. This is useful for tests.

func WithRequestEditorFn

func WithRequestEditorFn(fn RequestEditorFn) ClientOption

WithRequestEditorFn allows setting up a callback function, which will be called right before sending the request. This can be used to mutate the request.

type ClientWithResponses

type ClientWithResponses struct {
	ClientInterface
}

ClientWithResponses builds on ClientInterface to offer response payloads

func NewClientWithResponses

func NewClientWithResponses(server string, opts ...ClientOption) (*ClientWithResponses, error)

NewClientWithResponses creates a new ClientWithResponses, which wraps Client with return type handling

func (*ClientWithResponses) GenAudioToTextWithBodyWithResponse

func (c *ClientWithResponses) GenAudioToTextWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenAudioToTextResponse, error)

GenAudioToTextWithBodyWithResponse request with arbitrary body returning *GenAudioToTextResponse

func (*ClientWithResponses) GenImageToImageWithBodyWithResponse

func (c *ClientWithResponses) GenImageToImageWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToImageResponse, error)

GenImageToImageWithBodyWithResponse request with arbitrary body returning *GenImageToImageResponse

func (*ClientWithResponses) GenImageToTextWithBodyWithResponse

func (c *ClientWithResponses) GenImageToTextWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToTextResponse, error)

GenImageToTextWithBodyWithResponse request with arbitrary body returning *GenImageToTextResponse

func (*ClientWithResponses) GenImageToVideoWithBodyWithResponse

func (c *ClientWithResponses) GenImageToVideoWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToVideoResponse, error)

GenImageToVideoWithBodyWithResponse request with arbitrary body returning *GenImageToVideoResponse

func (*ClientWithResponses) GenLLMWithBodyWithResponse

func (c *ClientWithResponses) GenLLMWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenLLMResponse, error)

GenLLMWithBodyWithResponse request with arbitrary body returning *GenLLMResponse

func (*ClientWithResponses) GenLLMWithResponse

func (c *ClientWithResponses) GenLLMWithResponse(ctx context.Context, body GenLLMJSONRequestBody, reqEditors ...RequestEditorFn) (*GenLLMResponse, error)

func (*ClientWithResponses) GenLiveVideoToVideoWithBodyWithResponse

func (c *ClientWithResponses) GenLiveVideoToVideoWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenLiveVideoToVideoResponse, error)

GenLiveVideoToVideoWithBodyWithResponse request with arbitrary body returning *GenLiveVideoToVideoResponse

func (*ClientWithResponses) GenLiveVideoToVideoWithResponse

func (c *ClientWithResponses) GenLiveVideoToVideoWithResponse(ctx context.Context, body GenLiveVideoToVideoJSONRequestBody, reqEditors ...RequestEditorFn) (*GenLiveVideoToVideoResponse, error)

func (*ClientWithResponses) GenSegmentAnything2WithBodyWithResponse

func (c *ClientWithResponses) GenSegmentAnything2WithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenSegmentAnything2Response, error)

GenSegmentAnything2WithBodyWithResponse request with arbitrary body returning *GenSegmentAnything2Response

func (*ClientWithResponses) GenTextToImageWithBodyWithResponse

func (c *ClientWithResponses) GenTextToImageWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenTextToImageResponse, error)

GenTextToImageWithBodyWithResponse request with arbitrary body returning *GenTextToImageResponse

func (*ClientWithResponses) GenTextToImageWithResponse

func (c *ClientWithResponses) GenTextToImageWithResponse(ctx context.Context, body GenTextToImageJSONRequestBody, reqEditors ...RequestEditorFn) (*GenTextToImageResponse, error)

func (*ClientWithResponses) GenTextToSpeechWithBodyWithResponse

func (c *ClientWithResponses) GenTextToSpeechWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenTextToSpeechResponse, error)

GenTextToSpeechWithBodyWithResponse request with arbitrary body returning *GenTextToSpeechResponse

func (*ClientWithResponses) GenTextToSpeechWithResponse

func (c *ClientWithResponses) GenTextToSpeechWithResponse(ctx context.Context, body GenTextToSpeechJSONRequestBody, reqEditors ...RequestEditorFn) (*GenTextToSpeechResponse, error)

func (*ClientWithResponses) GenUpscaleWithBodyWithResponse

func (c *ClientWithResponses) GenUpscaleWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenUpscaleResponse, error)

GenUpscaleWithBodyWithResponse request with arbitrary body returning *GenUpscaleResponse

func (*ClientWithResponses) HardwareInfoWithResponse

func (c *ClientWithResponses) HardwareInfoWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HardwareInfoResponse, error)

HardwareInfoWithResponse request returning *HardwareInfoResponse

func (*ClientWithResponses) HardwareStatsWithResponse

func (c *ClientWithResponses) HardwareStatsWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HardwareStatsResponse, error)

HardwareStatsWithResponse request returning *HardwareStatsResponse

func (*ClientWithResponses) HealthWithResponse

func (c *ClientWithResponses) HealthWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HealthResponse, error)

HealthWithResponse request returning *HealthResponse

func (*ClientWithResponses) VersionWithResponse added in v0.8.6

func (c *ClientWithResponses) VersionWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*VersionResponse, error)

VersionWithResponse request returning *VersionResponse

type ClientWithResponsesInterface

type ClientWithResponsesInterface interface {
	// GenAudioToTextWithBodyWithResponse request with any body
	GenAudioToTextWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenAudioToTextResponse, error)

	// HardwareInfoWithResponse request
	HardwareInfoWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HardwareInfoResponse, error)

	// HardwareStatsWithResponse request
	HardwareStatsWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HardwareStatsResponse, error)

	// HealthWithResponse request
	HealthWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*HealthResponse, error)

	// GenImageToImageWithBodyWithResponse request with any body
	GenImageToImageWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToImageResponse, error)

	// GenImageToTextWithBodyWithResponse request with any body
	GenImageToTextWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToTextResponse, error)

	// GenImageToVideoWithBodyWithResponse request with any body
	GenImageToVideoWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenImageToVideoResponse, error)

	// GenLiveVideoToVideoWithBodyWithResponse request with any body
	GenLiveVideoToVideoWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenLiveVideoToVideoResponse, error)

	GenLiveVideoToVideoWithResponse(ctx context.Context, body GenLiveVideoToVideoJSONRequestBody, reqEditors ...RequestEditorFn) (*GenLiveVideoToVideoResponse, error)

	// GenLLMWithBodyWithResponse request with any body
	GenLLMWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenLLMResponse, error)

	GenLLMWithResponse(ctx context.Context, body GenLLMJSONRequestBody, reqEditors ...RequestEditorFn) (*GenLLMResponse, error)

	// GenSegmentAnything2WithBodyWithResponse request with any body
	GenSegmentAnything2WithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenSegmentAnything2Response, error)

	// GenTextToImageWithBodyWithResponse request with any body
	GenTextToImageWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenTextToImageResponse, error)

	GenTextToImageWithResponse(ctx context.Context, body GenTextToImageJSONRequestBody, reqEditors ...RequestEditorFn) (*GenTextToImageResponse, error)

	// GenTextToSpeechWithBodyWithResponse request with any body
	GenTextToSpeechWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenTextToSpeechResponse, error)

	GenTextToSpeechWithResponse(ctx context.Context, body GenTextToSpeechJSONRequestBody, reqEditors ...RequestEditorFn) (*GenTextToSpeechResponse, error)

	// GenUpscaleWithBodyWithResponse request with any body
	GenUpscaleWithBodyWithResponse(ctx context.Context, contentType string, body io.Reader, reqEditors ...RequestEditorFn) (*GenUpscaleResponse, error)

	// VersionWithResponse request
	VersionWithResponse(ctx context.Context, reqEditors ...RequestEditorFn) (*VersionResponse, error)
}

ClientWithResponsesInterface is the interface specification for the client with responses above.

type DockerClient

type DockerClient interface {
	ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, platform *ocispec.Platform, containerName string) (container.CreateResponse, error)
	ContainerInspect(ctx context.Context, containerID string) (types.ContainerJSON, error)
	ContainerList(ctx context.Context, options container.ListOptions) ([]types.Container, error)
	ContainerRemove(ctx context.Context, containerID string, options container.RemoveOptions) error
	ContainerStart(ctx context.Context, containerID string, options container.StartOptions) error
	ContainerStop(ctx context.Context, containerID string, options container.StopOptions) error
	ImageInspectWithRaw(ctx context.Context, imageID string) (types.ImageInspect, []byte, error)
	ImagePull(ctx context.Context, ref string, options image.PullOptions) (io.ReadCloser, error)
}

DockerClient is an interface for the Docker client, allowing for mocking in tests. NOTE: ensure any docker.Client methods used in this package are added.

func NewDefaultDockerClient added in v0.8.8

func NewDefaultDockerClient() (DockerClient, error)

type DockerManager

type DockerManager struct {
	// contains filtered or unexported fields
}

func NewDockerManager

func NewDockerManager(overrides ImageOverrides, verboseLogs bool, gpus []string, modelDir string, client DockerClient, containerCreatorID string) (*DockerManager, error)

func (*DockerManager) Borrow

func (m *DockerManager) Borrow(ctx context.Context, pipeline, modelID string) (*RunnerContainer, error)

func (*DockerManager) EnsureImageAvailable

func (m *DockerManager) EnsureImageAvailable(ctx context.Context, pipeline string, modelID string) error

EnsureImageAvailable ensures the container image is available locally for the given pipeline and model ID.

func (*DockerManager) GetCapacity added in v0.8.6

func (m *DockerManager) GetCapacity(pipeline, modelID string) (Capacity, int)

GetCapacity returns the current number of containers in use and idle It currently only supports a setup of a single model with the number of initial warm containers equalling max capacity. For example for Live AI we use this setup, we configure the number of warm containers to equal the max capacity we want to accept, all with the comfyui model.

func (*DockerManager) HardwareInformation added in v0.8.6

func (m *DockerManager) HardwareInformation() []HardwareInformation

func (*DockerManager) HasCapacity

func (m *DockerManager) HasCapacity(ctx context.Context, pipeline, modelID string) bool

HasCapacity checks if an unused managed container exists or if a GPU is available for a new container.

func (*DockerManager) Stop

func (m *DockerManager) Stop(ctx context.Context) error

func (*DockerManager) Version added in v0.8.6

func (m *DockerManager) Version() []Version

func (*DockerManager) Warm

func (m *DockerManager) Warm(ctx context.Context, pipeline string, modelID string, optimizationFlags OptimizationFlags) error

type EnvValue

type EnvValue string

EnvValue unmarshals JSON booleans as strings for compatibility with env variables.

func (EnvValue) String

func (sb EnvValue) String() string

String returns the string representation of the EnvValue.

func (*EnvValue) UnmarshalJSON

func (sb *EnvValue) UnmarshalJSON(b []byte) error

UnmarshalJSON converts JSON booleans to strings for EnvValue.

type GPUComputeInfo

type GPUComputeInfo struct {
	Id          string `json:"id"`
	Major       int    `json:"major"`
	MemoryFree  int    `json:"memory_free"`
	MemoryTotal int    `json:"memory_total"`
	Minor       int    `json:"minor"`
	Name        string `json:"name"`
}

GPUComputeInfo Model for detailed GPU compute information.

type GPUUtilizationInfo

type GPUUtilizationInfo struct {
	Id                 string `json:"id"`
	MemoryFree         int    `json:"memory_free"`
	MemoryTotal        int    `json:"memory_total"`
	Name               string `json:"name"`
	UtilizationCompute int    `json:"utilization_compute"`
	UtilizationMemory  int    `json:"utilization_memory"`
}

GPUUtilizationInfo Model for GPU utilization statistics.

type GenAudioToTextMultipartRequestBody

type GenAudioToTextMultipartRequestBody = BodyGenAudioToText

GenAudioToTextMultipartRequestBody defines body for GenAudioToText for multipart/form-data ContentType.

type GenAudioToTextResponse

type GenAudioToTextResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *TextResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON413      *HTTPError
	JSON415      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenAudioToTextResponse

func ParseGenAudioToTextResponse(rsp *http.Response) (*GenAudioToTextResponse, error)

ParseGenAudioToTextResponse parses an HTTP response from a GenAudioToTextWithResponse call

func (GenAudioToTextResponse) Status

func (r GenAudioToTextResponse) Status() string

Status returns HTTPResponse.Status

func (GenAudioToTextResponse) StatusCode

func (r GenAudioToTextResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenImageToImageMultipartRequestBody

type GenImageToImageMultipartRequestBody = BodyGenImageToImage

GenImageToImageMultipartRequestBody defines body for GenImageToImage for multipart/form-data ContentType.

type GenImageToImageResponse

type GenImageToImageResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *ImageResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenImageToImageResponse

func ParseGenImageToImageResponse(rsp *http.Response) (*GenImageToImageResponse, error)

ParseGenImageToImageResponse parses an HTTP response from a GenImageToImageWithResponse call

func (GenImageToImageResponse) Status

func (r GenImageToImageResponse) Status() string

Status returns HTTPResponse.Status

func (GenImageToImageResponse) StatusCode

func (r GenImageToImageResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenImageToTextMultipartRequestBody

type GenImageToTextMultipartRequestBody = BodyGenImageToText

GenImageToTextMultipartRequestBody defines body for GenImageToText for multipart/form-data ContentType.

type GenImageToTextResponse

type GenImageToTextResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *ImageToTextResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON413      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenImageToTextResponse

func ParseGenImageToTextResponse(rsp *http.Response) (*GenImageToTextResponse, error)

ParseGenImageToTextResponse parses an HTTP response from a GenImageToTextWithResponse call

func (GenImageToTextResponse) Status

func (r GenImageToTextResponse) Status() string

Status returns HTTPResponse.Status

func (GenImageToTextResponse) StatusCode

func (r GenImageToTextResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenImageToVideoMultipartRequestBody

type GenImageToVideoMultipartRequestBody = BodyGenImageToVideo

GenImageToVideoMultipartRequestBody defines body for GenImageToVideo for multipart/form-data ContentType.

type GenImageToVideoResponse

type GenImageToVideoResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *VideoResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenImageToVideoResponse

func ParseGenImageToVideoResponse(rsp *http.Response) (*GenImageToVideoResponse, error)

ParseGenImageToVideoResponse parses an HTTP response from a GenImageToVideoWithResponse call

func (GenImageToVideoResponse) Status

func (r GenImageToVideoResponse) Status() string

Status returns HTTPResponse.Status

func (GenImageToVideoResponse) StatusCode

func (r GenImageToVideoResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenLLMJSONRequestBody

type GenLLMJSONRequestBody = LLMRequest

GenLLMJSONRequestBody defines body for GenLLM for application/json ContentType.

type GenLLMResponse

type GenLLMResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *LLMResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenLLMResponse

func ParseGenLLMResponse(rsp *http.Response) (*GenLLMResponse, error)

ParseGenLLMResponse parses an HTTP response from a GenLLMWithResponse call

func (GenLLMResponse) Status

func (r GenLLMResponse) Status() string

Status returns HTTPResponse.Status

func (GenLLMResponse) StatusCode

func (r GenLLMResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenLiveVideoToVideoJSONRequestBody

type GenLiveVideoToVideoJSONRequestBody = LiveVideoToVideoParams

GenLiveVideoToVideoJSONRequestBody defines body for GenLiveVideoToVideo for application/json ContentType.

type GenLiveVideoToVideoResponse

type GenLiveVideoToVideoResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *LiveVideoToVideoResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenLiveVideoToVideoResponse

func ParseGenLiveVideoToVideoResponse(rsp *http.Response) (*GenLiveVideoToVideoResponse, error)

ParseGenLiveVideoToVideoResponse parses an HTTP response from a GenLiveVideoToVideoWithResponse call

func (GenLiveVideoToVideoResponse) Status

Status returns HTTPResponse.Status

func (GenLiveVideoToVideoResponse) StatusCode

func (r GenLiveVideoToVideoResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenSegmentAnything2MultipartRequestBody

type GenSegmentAnything2MultipartRequestBody = BodyGenSegmentAnything2

GenSegmentAnything2MultipartRequestBody defines body for GenSegmentAnything2 for multipart/form-data ContentType.

type GenSegmentAnything2Response

type GenSegmentAnything2Response struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *MasksResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenSegmentAnything2Response

func ParseGenSegmentAnything2Response(rsp *http.Response) (*GenSegmentAnything2Response, error)

ParseGenSegmentAnything2Response parses an HTTP response from a GenSegmentAnything2WithResponse call

func (GenSegmentAnything2Response) Status

Status returns HTTPResponse.Status

func (GenSegmentAnything2Response) StatusCode

func (r GenSegmentAnything2Response) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenTextToImageJSONRequestBody

type GenTextToImageJSONRequestBody = TextToImageParams

GenTextToImageJSONRequestBody defines body for GenTextToImage for application/json ContentType.

type GenTextToImageResponse

type GenTextToImageResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *ImageResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenTextToImageResponse

func ParseGenTextToImageResponse(rsp *http.Response) (*GenTextToImageResponse, error)

ParseGenTextToImageResponse parses an HTTP response from a GenTextToImageWithResponse call

func (GenTextToImageResponse) Status

func (r GenTextToImageResponse) Status() string

Status returns HTTPResponse.Status

func (GenTextToImageResponse) StatusCode

func (r GenTextToImageResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenTextToSpeechJSONRequestBody

type GenTextToSpeechJSONRequestBody = TextToSpeechParams

GenTextToSpeechJSONRequestBody defines body for GenTextToSpeech for application/json ContentType.

type GenTextToSpeechResponse

type GenTextToSpeechResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *AudioResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenTextToSpeechResponse

func ParseGenTextToSpeechResponse(rsp *http.Response) (*GenTextToSpeechResponse, error)

ParseGenTextToSpeechResponse parses an HTTP response from a GenTextToSpeechWithResponse call

func (GenTextToSpeechResponse) Status

func (r GenTextToSpeechResponse) Status() string

Status returns HTTPResponse.Status

func (GenTextToSpeechResponse) StatusCode

func (r GenTextToSpeechResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type GenUpscaleMultipartRequestBody

type GenUpscaleMultipartRequestBody = BodyGenUpscale

GenUpscaleMultipartRequestBody defines body for GenUpscale for multipart/form-data ContentType.

type GenUpscaleResponse

type GenUpscaleResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *ImageResponse
	JSON400      *HTTPError
	JSON401      *HTTPError
	JSON422      *HTTPValidationError
	JSON500      *HTTPError
}

func ParseGenUpscaleResponse

func ParseGenUpscaleResponse(rsp *http.Response) (*GenUpscaleResponse, error)

ParseGenUpscaleResponse parses an HTTP response from a GenUpscaleWithResponse call

func (GenUpscaleResponse) Status

func (r GenUpscaleResponse) Status() string

Status returns HTTPResponse.Status

func (GenUpscaleResponse) StatusCode

func (r GenUpscaleResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type HTTPError

type HTTPError struct {
	// Detail Detailed error information.
	Detail APIError `json:"detail"`
}

HTTPError HTTP error response model.

type HTTPValidationError

type HTTPValidationError struct {
	Detail *[]ValidationError `json:"detail,omitempty"`
}

HTTPValidationError defines model for HTTPValidationError.

type HardwareInfoResponse

type HardwareInfoResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *HardwareInformation
}

func ParseHardwareInfoResponse

func ParseHardwareInfoResponse(rsp *http.Response) (*HardwareInfoResponse, error)

ParseHardwareInfoResponse parses an HTTP response from a HardwareInfoWithResponse call

func (HardwareInfoResponse) Status

func (r HardwareInfoResponse) Status() string

Status returns HTTPResponse.Status

func (HardwareInfoResponse) StatusCode

func (r HardwareInfoResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type HardwareInformation

type HardwareInformation struct {
	GpuInfo  map[string]GPUComputeInfo `json:"gpu_info"`
	ModelId  string                    `json:"model_id"`
	Pipeline string                    `json:"pipeline"`
}

HardwareInformation Response model for GPU information.

type HardwareStats

type HardwareStats struct {
	GpuStats map[string]GPUUtilizationInfo `json:"gpu_stats"`
	ModelId  string                        `json:"model_id"`
	Pipeline string                        `json:"pipeline"`
}

HardwareStats Response model for real-time GPU statistics.

type HardwareStatsResponse

type HardwareStatsResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *HardwareStats
}

func ParseHardwareStatsResponse

func ParseHardwareStatsResponse(rsp *http.Response) (*HardwareStatsResponse, error)

ParseHardwareStatsResponse parses an HTTP response from a HardwareStatsWithResponse call

func (HardwareStatsResponse) Status

func (r HardwareStatsResponse) Status() string

Status returns HTTPResponse.Status

func (HardwareStatsResponse) StatusCode

func (r HardwareStatsResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type HealthCheck

type HealthCheck struct {
	// Status The health status of the pipeline
	Status HealthCheckStatus `json:"status"`
}

HealthCheck defines model for HealthCheck.

type HealthCheckStatus

type HealthCheckStatus string

HealthCheckStatus The health status of the pipeline

const (
	ERROR   HealthCheckStatus = "ERROR"
	IDLE    HealthCheckStatus = "IDLE"
	LOADING HealthCheckStatus = "LOADING"
	OK      HealthCheckStatus = "OK"
)

Defines values for HealthCheckStatus.

type HealthResponse

type HealthResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *HealthCheck
}

func ParseHealthResponse

func ParseHealthResponse(rsp *http.Response) (*HealthResponse, error)

ParseHealthResponse parses an HTTP response from a HealthWithResponse call

func (HealthResponse) Status

func (r HealthResponse) Status() string

Status returns HTTPResponse.Status

func (HealthResponse) StatusCode

func (r HealthResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type HttpRequestDoer

type HttpRequestDoer interface {
	Do(req *http.Request) (*http.Response, error)
}

Doer performs HTTP requests.

The standard http.Client implements this interface.

type ImageOverrides

type ImageOverrides struct {
	Default string            `json:"default"`
	Batch   map[string]string `json:"batch"`
	Live    map[string]string `json:"live"`
}

type ImageResponse

type ImageResponse struct {
	// Images The generated images.
	Images []Media `json:"images"`
}

ImageResponse Response model for image generation.

type ImageToTextResponse

type ImageToTextResponse struct {
	// Text The generated text.
	Text string `json:"text"`
}

ImageToTextResponse Response model for text generation.

type InvalidParamFormatError

type InvalidParamFormatError struct {
	ParamName string
	Err       error
}

func (*InvalidParamFormatError) Error

func (e *InvalidParamFormatError) Error() string

func (*InvalidParamFormatError) Unwrap

func (e *InvalidParamFormatError) Unwrap() error

type LLMChoice

type LLMChoice struct {
	Delta        *LLMMessage `json:"delta,omitempty"`
	FinishReason *string     `json:"finish_reason,omitempty"`
	Index        int         `json:"index"`
	Message      *LLMMessage `json:"message,omitempty"`
}

LLMChoice defines model for LLMChoice.

type LLMMessage

type LLMMessage struct {
	Content string `json:"content"`
	Role    string `json:"role"`
}

LLMMessage defines model for LLMMessage.

type LLMRequest

type LLMRequest struct {
	MaxTokens   *int         `json:"max_tokens,omitempty"`
	Messages    []LLMMessage `json:"messages"`
	Model       *string      `json:"model,omitempty"`
	Stream      *bool        `json:"stream,omitempty"`
	Temperature *float32     `json:"temperature,omitempty"`
	TopK        *int         `json:"top_k,omitempty"`
	TopP        *float32     `json:"top_p,omitempty"`
}

LLMRequest defines model for LLMRequest.

type LLMResponse

type LLMResponse struct {
	Choices []LLMChoice   `json:"choices"`
	Created int           `json:"created"`
	Id      string        `json:"id"`
	Model   string        `json:"model"`
	Usage   LLMTokenUsage `json:"usage"`
}

LLMResponse defines model for LLMResponse.

type LLMTokenUsage

type LLMTokenUsage struct {
	CompletionTokens int `json:"completion_tokens"`
	PromptTokens     int `json:"prompt_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

LLMTokenUsage defines model for LLMTokenUsage.

type LiveVideoToVideoParams

type LiveVideoToVideoParams struct {
	// ControlUrl URL for subscribing via Trickle protocol for updates in the live video-to-video generation params.
	ControlUrl *string `json:"control_url,omitempty"`

	// EventsUrl URL for publishing events via Trickle protocol for pipeline status and logs.
	EventsUrl *string `json:"events_url,omitempty"`

	// GatewayRequestId The ID of the Gateway request (for logging purposes).
	GatewayRequestId *string `json:"gateway_request_id,omitempty"`

	// ManifestId The manifest ID from the orchestrator (for logging purposes).
	ManifestId *string `json:"manifest_id,omitempty"`

	// ModelId Name of the pipeline to run in the live video to video job. Notice that this is named model_id for consistency with other routes, but it does not refer to a Hugging Face model ID. The exact model(s) depends on the pipeline implementation and might be configurable via the `params` argument.
	ModelId *string `json:"model_id,omitempty"`

	// Params Initial parameters for the pipeline.
	Params *map[string]interface{} `json:"params,omitempty"`

	// PublishUrl Destination URL of the outgoing stream to publish.
	PublishUrl string `json:"publish_url"`

	// StreamId The Stream ID (for logging purposes).
	StreamId *string `json:"stream_id,omitempty"`

	// SubscribeUrl Source URL of the incoming stream to subscribe to.
	SubscribeUrl string `json:"subscribe_url"`
}

LiveVideoToVideoParams defines model for LiveVideoToVideoParams.

type LiveVideoToVideoResponse

type LiveVideoToVideoResponse struct {
	// ControlUrl URL for updating the live video-to-video generation
	ControlUrl *string `json:"control_url,omitempty"`

	// EventsUrl URL for subscribing to events for pipeline status and logs
	EventsUrl *string `json:"events_url,omitempty"`

	// ManifestId Orchestrator manifest ID for this request
	ManifestId *string `json:"manifest_id,omitempty"`

	// PublishUrl Destination URL of the outgoing stream to publish to
	PublishUrl string `json:"publish_url"`

	// RequestId The ID generated for this request
	RequestId *string `json:"request_id,omitempty"`

	// SubscribeUrl Source URL of the incoming stream to subscribe to
	SubscribeUrl string `json:"subscribe_url"`
}

LiveVideoToVideoResponse Response model for live video-to-video generation.

type MasksResponse

type MasksResponse struct {
	// Logits The raw, unnormalized predictions (logits) for the masks.
	Logits string `json:"logits"`

	// Masks The generated masks.
	Masks string `json:"masks"`

	// Scores The model's confidence scores for each generated mask.
	Scores string `json:"scores"`
}

MasksResponse Response model for object segmentation.

type Media

type Media struct {
	// Nsfw Whether the media was flagged as NSFW.
	Nsfw bool `json:"nsfw"`

	// Seed The seed used to generate the media.
	Seed int `json:"seed"`

	// Url The URL where the media can be accessed.
	Url string `json:"url"`
}

Media A media object containing information about the generated media.

type MediaURL

type MediaURL struct {
	// Url The URL where the media can be accessed.
	Url string `json:"url"`
}

MediaURL A URL from which media can be accessed.

type MiddlewareFunc

type MiddlewareFunc func(http.Handler) http.Handler

type OptimizationFlags

type OptimizationFlags map[string]EnvValue

OptimizationFlags is a map of optimization flags to be passed to the pipeline.

type RequestEditorFn

type RequestEditorFn func(ctx context.Context, req *http.Request) error

RequestEditorFn is the function signature for the RequestEditor callback function

type RequiredHeaderError

type RequiredHeaderError struct {
	ParamName string
	Err       error
}

func (*RequiredHeaderError) Error

func (e *RequiredHeaderError) Error() string

func (*RequiredHeaderError) Unwrap

func (e *RequiredHeaderError) Unwrap() error

type RequiredParamError

type RequiredParamError struct {
	ParamName string
}

func (*RequiredParamError) Error

func (e *RequiredParamError) Error() string

type RunnerContainer

type RunnerContainer struct {
	RunnerContainerConfig
	Name     string
	Client   *ClientWithResponses
	Hardware *HardwareInformation
	Version  *Version

	BorrowCtx context.Context
	sync.RWMutex
}

func NewRunnerContainer

func NewRunnerContainer(ctx context.Context, cfg RunnerContainerConfig, name string) (rc *RunnerContainer, isLoading bool, err error)

type RunnerContainerConfig

type RunnerContainerConfig struct {
	Type             RunnerContainerType
	Pipeline         string
	ModelID          string
	Endpoint         RunnerEndpoint
	ContainerImageID string

	// For managed containers only
	ID                string
	GPU               string
	KeepWarm          bool
	OptimizationFlags OptimizationFlags
	// contains filtered or unexported fields
}

type RunnerContainerType

type RunnerContainerType int
const (
	Managed RunnerContainerType = iota
	External
)

type RunnerEndpoint

type RunnerEndpoint struct {
	URL   string
	Token string
}

type ServerInterface

type ServerInterface interface {
	// Audio To Text
	// (POST /audio-to-text)
	GenAudioToText(w http.ResponseWriter, r *http.Request)
	// Hardware Info
	// (GET /hardware/info)
	HardwareInfo(w http.ResponseWriter, r *http.Request)
	// Hardware Stats
	// (GET /hardware/stats)
	HardwareStats(w http.ResponseWriter, r *http.Request)
	// Health
	// (GET /health)
	Health(w http.ResponseWriter, r *http.Request)
	// Image To Image
	// (POST /image-to-image)
	GenImageToImage(w http.ResponseWriter, r *http.Request)
	// Image To Text
	// (POST /image-to-text)
	GenImageToText(w http.ResponseWriter, r *http.Request)
	// Image To Video
	// (POST /image-to-video)
	GenImageToVideo(w http.ResponseWriter, r *http.Request)
	// Live Video To Video
	// (POST /live-video-to-video)
	GenLiveVideoToVideo(w http.ResponseWriter, r *http.Request)
	// LLM
	// (POST /llm)
	GenLLM(w http.ResponseWriter, r *http.Request)
	// Segment Anything 2
	// (POST /segment-anything-2)
	GenSegmentAnything2(w http.ResponseWriter, r *http.Request)
	// Text To Image
	// (POST /text-to-image)
	GenTextToImage(w http.ResponseWriter, r *http.Request)
	// Text To Speech
	// (POST /text-to-speech)
	GenTextToSpeech(w http.ResponseWriter, r *http.Request)
	// Upscale
	// (POST /upscale)
	GenUpscale(w http.ResponseWriter, r *http.Request)
	// Version
	// (GET /version)
	Version(w http.ResponseWriter, r *http.Request)
}

ServerInterface represents all server handlers.

type ServerInterfaceWrapper

type ServerInterfaceWrapper struct {
	Handler            ServerInterface
	HandlerMiddlewares []MiddlewareFunc
	ErrorHandlerFunc   func(w http.ResponseWriter, r *http.Request, err error)
}

ServerInterfaceWrapper converts contexts to parameters.

func (*ServerInterfaceWrapper) GenAudioToText

func (siw *ServerInterfaceWrapper) GenAudioToText(w http.ResponseWriter, r *http.Request)

GenAudioToText operation middleware

func (*ServerInterfaceWrapper) GenImageToImage

func (siw *ServerInterfaceWrapper) GenImageToImage(w http.ResponseWriter, r *http.Request)

GenImageToImage operation middleware

func (*ServerInterfaceWrapper) GenImageToText

func (siw *ServerInterfaceWrapper) GenImageToText(w http.ResponseWriter, r *http.Request)

GenImageToText operation middleware

func (*ServerInterfaceWrapper) GenImageToVideo

func (siw *ServerInterfaceWrapper) GenImageToVideo(w http.ResponseWriter, r *http.Request)

GenImageToVideo operation middleware

func (*ServerInterfaceWrapper) GenLLM

GenLLM operation middleware

func (*ServerInterfaceWrapper) GenLiveVideoToVideo

func (siw *ServerInterfaceWrapper) GenLiveVideoToVideo(w http.ResponseWriter, r *http.Request)

GenLiveVideoToVideo operation middleware

func (*ServerInterfaceWrapper) GenSegmentAnything2

func (siw *ServerInterfaceWrapper) GenSegmentAnything2(w http.ResponseWriter, r *http.Request)

GenSegmentAnything2 operation middleware

func (*ServerInterfaceWrapper) GenTextToImage

func (siw *ServerInterfaceWrapper) GenTextToImage(w http.ResponseWriter, r *http.Request)

GenTextToImage operation middleware

func (*ServerInterfaceWrapper) GenTextToSpeech

func (siw *ServerInterfaceWrapper) GenTextToSpeech(w http.ResponseWriter, r *http.Request)

GenTextToSpeech operation middleware

func (*ServerInterfaceWrapper) GenUpscale

func (siw *ServerInterfaceWrapper) GenUpscale(w http.ResponseWriter, r *http.Request)

GenUpscale operation middleware

func (*ServerInterfaceWrapper) HardwareInfo

func (siw *ServerInterfaceWrapper) HardwareInfo(w http.ResponseWriter, r *http.Request)

HardwareInfo operation middleware

func (*ServerInterfaceWrapper) HardwareStats

func (siw *ServerInterfaceWrapper) HardwareStats(w http.ResponseWriter, r *http.Request)

HardwareStats operation middleware

func (*ServerInterfaceWrapper) Health

Health operation middleware

func (*ServerInterfaceWrapper) Version added in v0.8.6

Version operation middleware

type TextResponse

type TextResponse struct {
	// Chunks The generated text chunks.
	Chunks []Chunk `json:"chunks"`

	// Text The generated text.
	Text string `json:"text"`
}

TextResponse Response model for text generation.

type TextToImageParams

type TextToImageParams struct {
	// GuidanceScale Encourages model to generate images closely linked to the text prompt (higher values may reduce image quality).
	GuidanceScale *float32 `json:"guidance_scale,omitempty"`

	// Height The height in pixels of the generated image.
	Height *int `json:"height,omitempty"`

	// Loras A LoRA (Low-Rank Adaptation) model and its corresponding weight for image generation. Example: { "latent-consistency/lcm-lora-sdxl": 1.0, "nerijs/pixel-art-xl": 1.2}.
	Loras *string `json:"loras,omitempty"`

	// ModelId Hugging Face model ID used for image generation.
	ModelId *string `json:"model_id,omitempty"`

	// NegativePrompt Text prompt(s) to guide what to exclude from image generation. Ignored if guidance_scale < 1.
	NegativePrompt *string `json:"negative_prompt,omitempty"`

	// NumImagesPerPrompt Number of images to generate per prompt.
	NumImagesPerPrompt *int `json:"num_images_per_prompt,omitempty"`

	// NumInferenceSteps Number of denoising steps. More steps usually lead to higher quality images but slower inference. Modulated by strength.
	NumInferenceSteps *int `json:"num_inference_steps,omitempty"`

	// Prompt Text prompt(s) to guide image generation. Separate multiple prompts with '|' if supported by the model.
	Prompt string `json:"prompt"`

	// SafetyCheck Perform a safety check to estimate if generated images could be offensive or harmful.
	SafetyCheck *bool `json:"safety_check,omitempty"`

	// Seed Seed for random number generation.
	Seed *int `json:"seed,omitempty"`

	// Width The width in pixels of the generated image.
	Width *int `json:"width,omitempty"`
}

TextToImageParams defines model for TextToImageParams.

type TextToSpeechParams

type TextToSpeechParams struct {
	// Description Description of speaker to steer text to speech generation.
	Description *string `json:"description,omitempty"`

	// ModelId Hugging Face model ID used for text to speech generation.
	ModelId *string `json:"model_id,omitempty"`

	// Text Text input for speech generation.
	Text *string `json:"text,omitempty"`
}

TextToSpeechParams defines model for TextToSpeechParams.

type TooManyValuesForParamError

type TooManyValuesForParamError struct {
	ParamName string
	Count     int
}

func (*TooManyValuesForParamError) Error

type UnescapedCookieParamError

type UnescapedCookieParamError struct {
	ParamName string
	Err       error
}

func (*UnescapedCookieParamError) Error

func (e *UnescapedCookieParamError) Error() string

func (*UnescapedCookieParamError) Unwrap

func (e *UnescapedCookieParamError) Unwrap() error

type Unimplemented

type Unimplemented struct{}

func (Unimplemented) GenAudioToText

func (_ Unimplemented) GenAudioToText(w http.ResponseWriter, r *http.Request)

Audio To Text (POST /audio-to-text)

func (Unimplemented) GenImageToImage

func (_ Unimplemented) GenImageToImage(w http.ResponseWriter, r *http.Request)

Image To Image (POST /image-to-image)

func (Unimplemented) GenImageToText

func (_ Unimplemented) GenImageToText(w http.ResponseWriter, r *http.Request)

Image To Text (POST /image-to-text)

func (Unimplemented) GenImageToVideo

func (_ Unimplemented) GenImageToVideo(w http.ResponseWriter, r *http.Request)

Image To Video (POST /image-to-video)

func (Unimplemented) GenLLM

func (_ Unimplemented) GenLLM(w http.ResponseWriter, r *http.Request)

LLM (POST /llm)

func (Unimplemented) GenLiveVideoToVideo

func (_ Unimplemented) GenLiveVideoToVideo(w http.ResponseWriter, r *http.Request)

Live Video To Video (POST /live-video-to-video)

func (Unimplemented) GenSegmentAnything2

func (_ Unimplemented) GenSegmentAnything2(w http.ResponseWriter, r *http.Request)

Segment Anything 2 (POST /segment-anything-2)

func (Unimplemented) GenTextToImage

func (_ Unimplemented) GenTextToImage(w http.ResponseWriter, r *http.Request)

Text To Image (POST /text-to-image)

func (Unimplemented) GenTextToSpeech

func (_ Unimplemented) GenTextToSpeech(w http.ResponseWriter, r *http.Request)

Text To Speech (POST /text-to-speech)

func (Unimplemented) GenUpscale

func (_ Unimplemented) GenUpscale(w http.ResponseWriter, r *http.Request)

Upscale (POST /upscale)

func (Unimplemented) HardwareInfo

func (_ Unimplemented) HardwareInfo(w http.ResponseWriter, r *http.Request)

Hardware Info (GET /hardware/info)

func (Unimplemented) HardwareStats

func (_ Unimplemented) HardwareStats(w http.ResponseWriter, r *http.Request)

Hardware Stats (GET /hardware/stats)

func (Unimplemented) Health

func (_ Unimplemented) Health(w http.ResponseWriter, r *http.Request)

Health (GET /health)

func (Unimplemented) Version added in v0.8.6

func (_ Unimplemented) Version(w http.ResponseWriter, r *http.Request)

Version (GET /version)

type UnmarshalingParamError

type UnmarshalingParamError struct {
	ParamName string
	Err       error
}

func (*UnmarshalingParamError) Error

func (e *UnmarshalingParamError) Error() string

func (*UnmarshalingParamError) Unwrap

func (e *UnmarshalingParamError) Unwrap() error

type ValidationError

type ValidationError struct {
	Loc  []ValidationError_Loc_Item `json:"loc"`
	Msg  string                     `json:"msg"`
	Type string                     `json:"type"`
}

ValidationError defines model for ValidationError.

type ValidationErrorLoc0

type ValidationErrorLoc0 = string

ValidationErrorLoc0 defines model for .

type ValidationErrorLoc1

type ValidationErrorLoc1 = int

ValidationErrorLoc1 defines model for .

type ValidationError_Loc_Item

type ValidationError_Loc_Item struct {
	// contains filtered or unexported fields
}

ValidationError_Loc_Item defines model for ValidationError.loc.Item.

func (ValidationError_Loc_Item) AsValidationErrorLoc0

func (t ValidationError_Loc_Item) AsValidationErrorLoc0() (ValidationErrorLoc0, error)

AsValidationErrorLoc0 returns the union data inside the ValidationError_Loc_Item as a ValidationErrorLoc0

func (ValidationError_Loc_Item) AsValidationErrorLoc1

func (t ValidationError_Loc_Item) AsValidationErrorLoc1() (ValidationErrorLoc1, error)

AsValidationErrorLoc1 returns the union data inside the ValidationError_Loc_Item as a ValidationErrorLoc1

func (*ValidationError_Loc_Item) FromValidationErrorLoc0

func (t *ValidationError_Loc_Item) FromValidationErrorLoc0(v ValidationErrorLoc0) error

FromValidationErrorLoc0 overwrites any union data inside the ValidationError_Loc_Item as the provided ValidationErrorLoc0

func (*ValidationError_Loc_Item) FromValidationErrorLoc1

func (t *ValidationError_Loc_Item) FromValidationErrorLoc1(v ValidationErrorLoc1) error

FromValidationErrorLoc1 overwrites any union data inside the ValidationError_Loc_Item as the provided ValidationErrorLoc1

func (ValidationError_Loc_Item) MarshalJSON

func (t ValidationError_Loc_Item) MarshalJSON() ([]byte, error)

func (*ValidationError_Loc_Item) MergeValidationErrorLoc0

func (t *ValidationError_Loc_Item) MergeValidationErrorLoc0(v ValidationErrorLoc0) error

MergeValidationErrorLoc0 performs a merge with any union data inside the ValidationError_Loc_Item, using the provided ValidationErrorLoc0

func (*ValidationError_Loc_Item) MergeValidationErrorLoc1

func (t *ValidationError_Loc_Item) MergeValidationErrorLoc1(v ValidationErrorLoc1) error

MergeValidationErrorLoc1 performs a merge with any union data inside the ValidationError_Loc_Item, using the provided ValidationErrorLoc1

func (*ValidationError_Loc_Item) UnmarshalJSON

func (t *ValidationError_Loc_Item) UnmarshalJSON(b []byte) error

type Version added in v0.8.6

type Version struct {
	ModelId  string `json:"model_id"`
	Pipeline string `json:"pipeline"`

	// Version The version of the Runner
	Version string `json:"version"`
}

Version defines model for Version.

type VersionResponse added in v0.8.6

type VersionResponse struct {
	Body         []byte
	HTTPResponse *http.Response
	JSON200      *Version
}

func ParseVersionResponse added in v0.8.6

func ParseVersionResponse(rsp *http.Response) (*VersionResponse, error)

ParseVersionResponse parses an HTTP response from a VersionWithResponse call

func (VersionResponse) Status added in v0.8.6

func (r VersionResponse) Status() string

Status returns HTTPResponse.Status

func (VersionResponse) StatusCode added in v0.8.6

func (r VersionResponse) StatusCode() int

StatusCode returns HTTPResponse.StatusCode

type VideoResponse

type VideoResponse struct {
	// Frames The generated video frames.
	Frames [][]Media `json:"frames"`
}

VideoResponse Response model for video generation.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

func NewWorker

func NewWorker(imageOverrides ImageOverrides, verboseLogs bool, gpus []string, modelDir string, containerCreatorID string) (*Worker, error)

func (*Worker) AudioToText

func (*Worker) EnsureImageAvailable

func (w *Worker) EnsureImageAvailable(ctx context.Context, pipeline string, modelID string) error

func (*Worker) GetLiveAICapacity added in v0.8.6

func (w *Worker) GetLiveAICapacity(pipeline, modelID string) Capacity

func (*Worker) HardwareInformation

func (w *Worker) HardwareInformation() []HardwareInformation

func (*Worker) HasCapacity

func (w *Worker) HasCapacity(pipeline, modelID string) bool

HasCapacity returns true if the worker has capacity for the given pipeline and model ID.

func (*Worker) ImageToImage

func (*Worker) ImageToVideo

func (*Worker) LLM

func (w *Worker) LLM(ctx context.Context, req GenLLMJSONRequestBody) (interface{}, error)

func (*Worker) SegmentAnything2

func (*Worker) Stop

func (w *Worker) Stop(ctx context.Context) error

func (*Worker) TextToImage

func (*Worker) TextToSpeech

func (*Worker) Upscale

func (*Worker) Version added in v0.8.6

func (w *Worker) Version() []Version

func (*Worker) Warm

func (w *Worker) Warm(ctx context.Context, pipeline string, modelID string, endpoint RunnerEndpoint, optimizationFlags OptimizationFlags) error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL