Documentation
¶
Overview ¶
Package hf contains HuggingFace integration logic for AIS.
- Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.
Package hf contains HuggingFace integration logic for AIS.
- Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.
Package hf contains HuggingFace integration logic for AIS.
- Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.
Index ¶
- Constants
- func BuildHuggingFaceURL(model, dataset, file, revision string) (string, error)
- func ExtractDatasetFromHFMarker(marker string) (string, error)
- func ExtractFileName(url string) string
- func ExtractModelFromHFMarker(marker string) (string, error)
- func ExtractParquetFiles(data map[string]any) cos.StrKVs
- func GetHFDatasetParquetFiles(dataset, token string) (cos.StrKVs, error)
- func GetHFModelFiles(model, token string) (cos.StrKVs, error)
- func HasHuggingFaceRepoFlags(hasModel, hasDataset bool) bool
- func IsHuggingFaceURL(rawURL string) bool
- type FileInfo
- type HFModelFile
- type HFModelResponse
Constants ¶
const (
HfFullRepoMarker = "HF_FULL_REPO_DOWNLOAD:"
)
HuggingFace URL patterns and constants
Variables ¶
This section is empty.
Functions ¶
func BuildHuggingFaceURL ¶
BuildHuggingFaceURL constructs a HuggingFace download URL from parameters
func ExtractDatasetFromHFMarker ¶
ExtractDatasetFromHFMarker parses dataset name from hfFullRepoMarker
func ExtractFileName ¶
ExtractFileName extracts filename from HuggingFace URL
func ExtractModelFromHFMarker ¶
ExtractModelFromHFMarker parses model name from hfFullRepoMarker
func ExtractParquetFiles ¶
ExtractParquetFiles parses the JSON structure from HF API
func GetHFDatasetParquetFiles ¶
GetHFDatasetParquetFiles fetches parquet file mappings for a HuggingFace dataset
func GetHFModelFiles ¶
GetHFModelFiles fetches all files in a HuggingFace model repository
func HasHuggingFaceRepoFlags ¶
HasHuggingFaceRepoFlags checks if only ModelFlag OR DatasetFlag (not both) is set
func IsHuggingFaceURL ¶
IsHuggingFaceURL checks if a URL is a HuggingFace URL
Types ¶
type HFModelFile ¶
type HFModelFile struct {
Filename string `json:"rfilename"`
}
HFModelFile represents a file in HuggingFace model repository
type HFModelResponse ¶
type HFModelResponse struct {
Siblings []HFModelFile `json:"siblings"`
}
HFModelResponse represents the HuggingFace model API response
Source Files