Documentation
¶
Overview ¶
Package modelpull pulls and stores AI model artifacts on the node.
Index ¶
- Constants
- func CleanupStaleIncomplete(root string, maxAge time.Duration, now time.Time) (int, error)
- func ContentDir(modelsRoot, name string) string
- func ContentStagingDir(modelsRoot, name string) string
- func EnsureDiskSpace(needed, free, total, thresholdPercent int64, location string) error
- func IsHFCommit(revision string) bool
- func IsOCIDigest(revision string) bool
- func KnowledgeRoot(diskDirectory string) string
- func ManifestPath(modelsRoot, name string) string
- func ModelDir(modelsRoot, name string) string
- func OCIStoreDir(modelsRoot string) string
- func Root(diskDirectory string) string
- func WatchReader(r io.Reader, already, total int64, report func(downloaded, total int64)) io.Reader
- func WriteOnDiskManifest(path string, m OnDiskManifest) error
- type DiskGuard
- type HFRef
- type OCIRef
- type OnDiskManifest
- type Request
- type Result
Constants ¶
const ( OCIStoreDirName = "oci-store" ContentDirName = "content" ManifestFile = "manifest.json" // IncompleteExt marks a payload that has not been atomically committed. IncompleteExt = ".incomplete" // ContentStagingSuffix is the in-progress content directory next to content/. ContentStagingSuffix = ".tmp" // StaleIncompleteAge is how long a partial download may sit before cleanup. StaleIncompleteAge = 7 * 24 * time.Hour )
const ( RevisionKindTag = "tag" RevisionKindDigest = "digest" RevisionKindCommit = "commit" RevisionKindBranch = "branch" )
Variables ¶
This section is empty.
Functions ¶
func CleanupStaleIncomplete ¶
CleanupStaleIncomplete removes leftover partial downloads older than maxAge.
func ContentDir ¶
ContentDir returns {modelsRoot}/{metadata.name}/content.
func ContentStagingDir ¶
ContentStagingDir returns the in-progress content directory for one model.
func EnsureDiskSpace ¶
EnsureDiskSpace fails when needed bytes cannot be stored while keeping thresholdPercent of total capacity free under location.
func IsHFCommit ¶
IsHFCommit reports whether revision is a 40-character git commit pin.
func IsOCIDigest ¶
IsOCIDigest reports whether revision is a sha256 digest pin.
func KnowledgeRoot ¶
KnowledgeRoot returns {diskDirectory}/knowledge.
func ManifestPath ¶
ManifestPath returns {modelsRoot}/{metadata.name}/manifest.json.
func OCIStoreDir ¶
OCIStoreDir returns {modelsRoot}/oci-store.
func WatchReader ¶
WatchReader wraps r and reports cumulative downloaded bytes as data is copied.
func WriteOnDiskManifest ¶
func WriteOnDiskManifest(path string, m OnDiskManifest) error
WriteOnDiskManifest writes manifest.json atomically next to the materialized content.
Types ¶
type DiskGuard ¶
DiskGuard rejects a pull when the estimated artifact would leave too little free space.
type HFRef ¶
type HFRef struct {
// Revision is the value sent to the Hub (main, branch, tag, or commit).
Revision string
// Floating is true for branches/tags (including the empty→main default).
Floating bool
// Kind is "commit" or "branch".
Kind string
// Requested is the original revision string after trim.
Requested string
}
HFRef is a resolved Hugging Face Hub revision derived from spec.revision.
func ResolveHFRevision ¶
ResolveHFRevision maps a model revision onto a Hub revision parameter. An empty revision becomes "main". A 40-character hex string is a commit pin.
type OCIRef ¶
type OCIRef struct {
// Repo is the repository path. A registry host, when present, is resolved by the pull adapter.
Repo string
// Reference is the tag or digest passed to the registry (e.g. "latest" or "sha256:…").
Reference string
// Tag is the store index tag (repo:tag or repo@sha256:…).
Tag string
// Floating is true for tags (including latest) and false for digest pins.
Floating bool
// Kind is "tag" or "digest".
Kind string
// Requested is the original revision string after trim.
Requested string
// ResolvedRevision is the revision recorded after resolve (latest when empty).
ResolvedRevision string
}
OCIRef is a resolved OCI artifact reference derived from spec.repo and spec.revision.
func ResolveOCIRevision ¶
ResolveOCIRevision maps a model revision onto an OCI tag or digest reference. An empty revision becomes "latest". A sha256:<64 hex> revision is a digest pin.
type OnDiskManifest ¶
type OnDiskManifest struct {
MetadataName string `json:"metadataName"`
RegistryID int `json:"registryId"`
RegistryType string `json:"registryType"`
Repo string `json:"repo"`
RequestedRevision string `json:"requestedRevision"`
ResolvedRevision string `json:"resolvedRevision"`
Digest string `json:"digest"`
Format string `json:"format"`
Files []string `json:"files"`
ContentPaths []string `json:"contentPaths"`
Blobs []string `json:"blobs,omitempty"`
TotalBytes int64 `json:"totalBytes"`
RevisionFloating bool `json:"revisionFloating"`
RevisionKind string `json:"revisionKind"`
PulledAt string `json:"pulledAt"`
OCIModelConfig json.RawMessage `json:"ociModelConfig,omitempty"`
IdentityKey string `json:"identityKey,omitempty"`
}
OnDiskManifest is the per-model state file written to {metadata.name}/manifest.json.
func ReadOnDiskManifest ¶
func ReadOnDiskManifest(path string) (OnDiskManifest, error)
ReadOnDiskManifest loads {metadata.name}/manifest.json when present.
type Request ¶
type Request struct {
// Name is metadata.name and the on-disk directory under the models root.
Name string
// Repo is spec.repo. A registry host may be included; the pull adapter resolves it.
Repo string
// Revision is spec.revision (tag, digest, or empty).
Revision string
// Registry is the source registry row. Type must match the pull adapter.
Registry *models.Registry
// Files is spec.files (Hub pulls only; ignored for OCI artifacts).
Files []string
// ModelsRoot is {diskDirectory}/models.
ModelsRoot string
// FormatHint is optional spec.format.
FormatHint string
// Disk, when set, is consulted after size is known and before payload download.
Disk DiskGuard
// OnProgress reports bytes copied so far (and total when known).
OnProgress func(downloaded, total int64)
}
Request is the input for an artifact pull.
func (Request) ReportProgress ¶
ReportProgress invokes OnProgress when a callback is set.
type Result ¶
type Result struct {
Digest string
ResolvedRevision string
RevisionFloating bool
RevisionKind string
Format string
ContentPaths []string
TotalBytes int64
ManifestPath string
ContentPath string
OCIModelConfig json.RawMessage
Blobs []string
}
Result is the outcome of a successful pull and materialize.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package format detects and unpacks OCI AI artifact packaging formats.
|
Package format detects and unpacks OCI AI artifact packaging formats. |
|
Package hf pulls AI model artifacts from Hugging Face Hub registries.
|
Package hf pulls AI model artifacts from Hugging Face Hub registries. |
|
Package oci pulls AI model artifacts from OCI registries.
|
Package oci pulls AI model artifacts from OCI registries. |
|
Package ocistore is the shared on-disk store for OCI model artifacts.
|
Package ocistore is the shared on-disk store for OCI model artifacts. |