Documentation
¶
Index ¶
- func FetchModelsByPlayground(ctx context.Context, k8sClient client.Client, ...) (models []*coreapi.OpenModel, err error)
- func FetchModelsByService(ctx context.Context, k8sClient client.Client, service *inferenceapi.Service) (models []*coreapi.OpenModel, err error)
- type BackendRuntimeParser
- func (p *BackendRuntimeParser) Args(mode InferenceMode, models []*coreapi.OpenModel) ([]string, error)
- func (p *BackendRuntimeParser) Commands() []string
- func (p *BackendRuntimeParser) Envs() []corev1.EnvVar
- func (p *BackendRuntimeParser) Image(version string) string
- func (p *BackendRuntimeParser) Resources() inferenceapi.ResourceRequirements
- func (p *BackendRuntimeParser) Version() string
- type InferenceMode
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FetchModelsByPlayground ¶
func FetchModelsByPlayground(ctx context.Context, k8sClient client.Client, playground *inferenceapi.Playground) (models []*coreapi.OpenModel, err error)
Types ¶
type BackendRuntimeParser ¶
type BackendRuntimeParser struct {
// contains filtered or unexported fields
}
TODO: add unit tests.
func NewBackendRuntimeParser ¶
func NewBackendRuntimeParser(backendRuntime *inferenceapi.BackendRuntime) *BackendRuntimeParser
func (*BackendRuntimeParser) Args ¶
func (p *BackendRuntimeParser) Args(mode InferenceMode, models []*coreapi.OpenModel) ([]string, error)
func (*BackendRuntimeParser) Commands ¶
func (p *BackendRuntimeParser) Commands() []string
func (*BackendRuntimeParser) Envs ¶
func (p *BackendRuntimeParser) Envs() []corev1.EnvVar
func (*BackendRuntimeParser) Image ¶
func (p *BackendRuntimeParser) Image(version string) string
func (*BackendRuntimeParser) Resources ¶
func (p *BackendRuntimeParser) Resources() inferenceapi.ResourceRequirements
func (*BackendRuntimeParser) Version ¶
func (p *BackendRuntimeParser) Version() string
type InferenceMode ¶
type InferenceMode string
const ( DefaultInferenceMode InferenceMode = "default" SpeculativeDecodingInferenceMode InferenceMode = "speculative-decoding" )
These two modes are preset.
func PlaygroundInferenceMode ¶
func PlaygroundInferenceMode(playground *inferenceapi.Playground) InferenceMode
PlaygroundInferenceMode gets the mode of inference process, supports default or speculative-decoding for now, which is aligned with backendRuntime.
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.