hub

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: Apache-2.0 Imports: 21 Imported by: 17

README

hub package

Downloads HuggingFace Hub files, a port of huggingFace_hub python library to Go.

Introduction

A simple, straight-forward port of github.com/huggingface/huggingface_hub library for Go.

Features supported:

  • Cache system that matches HuggingFace Hub, so the same cache can be shared with Python.
  • Concurrency safe: only one download when multiple workers are trying to download simultaneously the same model.
  • Allow arbitrary progress function to be called (for progress bar).
  • Arbitrary revision.
  • Parallel download of files, max=20 by default.

TODOs:

  • Add support for optional parameters.
  • Authentication tokens: should be relatively easy.
  • Resume downloads from interrupted connections.
  • Check disk-space before starting to download.

Example

Enumerate files from a HuggingFace repository and download all of them to a cache.

	repo := hub.New(modelID).WithAuth(hfAuthToken)
	var fileNames []string
	for fileName, err := range repo.IterFileNames() {
		if err != nil { panic(err) }
		fmt.Printf("\t%s\n", fileName)
		fileNames = append(fileNames, fileName)
	}
	downloadedFiles, err := repo.DownloadFiles(fileNames...)
	if err != nil { ... }

Documentation

Overview

Package hub can be used to download and cache files from HuggingFace Hub, which may be models, tokenizers or anything.

It is meant to be a port of huggingFace_hub python library to Go, and be able to share the same cache structure (usually under "~/.cache/huggingface/hub").

It is also safe to be used concurrently by multiple programs -- it uses file system lock to control concurrency.

Typical usage will be something like: 

repo := hub.New(modelID).WithAuth(hfAuthToken)
var fileNames []string
for fileName, err := range repo.IterFileNames() {
	if err != nil { panic(err) }
	fmt.Printf("\t%s\n", fileName)
	fileNames = append(fileNames, fileName)
}
downloadedFiles, err := repo.DownloadFiles(fileNames...)
if err != nil { ... }

From here, downloadedFiles will point to files in the local cache that one can read.

Environment variables:

- HF_ENDPOINT: Where to connect to huggingface, default is https://huggingface.co - XDG_CACHE_HOME: Cache directory, defaults to ${HOME}/.cache

Index

Constants

View Source 
const (
	HeaderXRepoCommit = "X-Repo-Commit"
	HeaderXLinkedETag = "X-Linked-Etag"
	HeaderXLinkedSize = "X-Linked-Size"
)
View Source 
const RepoIdSeparator = "--"

RepoIdSeparator is used to separate repository/model names parts when mapping to file names. Likely only for internal use.

Variables

View Source 
var (
	// DefaultDirCreationPerm is used when creating new cache subdirectories.
	DefaultDirCreationPerm = os.FileMode(0755)

	// DefaultFileCreationPerm is used when creating files inside the cache subdirectories.
	DefaultFileCreationPerm = os.FileMode(0644)
)
View Source 
var SessionId string

SessionId is unique and always created anew at the start of the program, and used during the life of the program.

Functions

func DefaultCacheDir 

func DefaultCacheDir() string

DefaultCacheDir for HuggingFace Hub, same used by the python library.

Its prefix is either `${XDG_CACHE_HOME}` if set, or `~/.cache` otherwise. Followed by `/huggingface/hub/`. So typically: `~/.cache/huggingface/hub/`.

func DefaultHttpUserAgent 

func DefaultHttpUserAgent() string

DefaultHttpUserAgent returns a user agent to use with HuggingFace Hub API.

Types

type CardData added in v0.4.0 

type CardData struct {
	LibraryName string `json:"library_name"`
	License     string `json:"license"`
	LicenseLink string `json:"license_link"`
	PipelineTag string `json:"pipeline_tag"`
	BaseModel   any    `json:"base_model"` // Can be a string, or slice of strings, or nil.
}

CardData contains metadata about the model card.

type FileInfo 

type FileInfo struct {
	Name string `json:"rfilename"`

	// Size is the file size in bytes.
	//
	// It may be left as 0 if not retrieved (if set WithExtraBlobInfo(false)).
	Size int64 `json:"size"`

	// BlobID is This is the standard Git Object ID (a SHA-1 hash) representing the file in the Git tree.
	//
	// It may be left as "" if not retrieved (if set WithExtraBlobsInfo(false)).
	BlobID string `json:"blobId"`

	// LFS holds Git LFS details if the file is tracked by LFS.
	//
	// It is not retrieved if WithExtraBlobsInfo(false).
	LFS *LFSInfo `json:"lfs"`
}

FileInfo represents one of the model files, in the Info structure.

The Hub is built on top of Git. Because standard Git is highly inefficient at handling large files (like machine learning model weights or massive datasets), Hugging Face heavily utilizes Git LFS (Large File Storage).

type LFSInfo added in v0.4.0 

type LFSInfo struct {
	SHA256 string `json:"sha256"`
	Size   int64  `json:"size"`

	// PointerSize the size of the tiny Git LFS pointer file that lives inside the Git tree, in bytes. Because it only
	// contains a few lines of text (the LFS version, the SHA-256 hash of the real file, and the real file's size), this
	// value is almost always between 130 and 135 bytes. If a file is a standard text file and not tracked by LFS, this
	// concept doesn't apply (it may be omitted or set to 0).
	PointerSize int64 `json:"pointerSize"`
}

LFSInfo holds Git LFS details for a file tracked by LFS.

type Repo 

type Repo struct {
	// ID of the Repo may include owner/model. E.g.: google/gemma-2-2b-it
	ID string

	// Verbosity: 0 for quiet operation; 1 for information about progress; 2 and higher for debugging.
	Verbosity int

	// MaxParallelDownload indicates how many files to download at the same time. Default is 20.
	// If set to <= 0 it will download all files in parallel.
	// Set to 1 to make downloads sequential.
	MaxParallelDownload int
	// contains filtered or unexported fields
}

Repo from which one wants to download files. Create it with New.

func New 

func New(id string) *Repo

New creates a reference to a HuggingFace model given its id.

It uses the default cache directory in ${XDG_CACHE_HOME} (if set) or `~/.cache`, in a format that is shared with huggingface-hub for python library. The cache is share across various programs, including Python programs. Use Repo.WithCacheDir to change it, or NewWithDir to use a plain directory structure, that is not shared across programs.

The id typically include owner/model. E.g.: "google/gemma-2-2b-it"

It defaults to being a RepoTypeModel repository. But you can change it with Repo.WithType.

If authentication is needed, use Repo.WithAuth.

func (*Repo) CacheDir added in v0.3.5 

func (r *Repo) CacheDir() (string, error)

CacheDir returns the cache subdirectory for the repository. It creates the directory if it doesn't exist.

func (*Repo) DownloadFile 

func (r *Repo) DownloadFile(file string) (downloadedPath string, err error)

DownloadFile is a shortcut to DownloadFiles with only one file.

func (*Repo) DownloadFileCtx added in v0.3.2 

func (r *Repo) DownloadFileCtx(ctx context.Context, file string) (downloadedPath string, err error)

DownloadFileCtx is like DownloadFile but accepts a context for cancellation support.

func (*Repo) DownloadFiles 

func (r *Repo) DownloadFiles(repoFiles ...string) (downloadedPaths []string, err error)

DownloadFiles downloads the repository files (the names returned by repo.IterFileNames), and return the path to the downloaded files in the cache structure.

The returned downloadPaths can be read, but shouldn't be modified, since there may be other programs using the same files.

func (*Repo) DownloadFilesCtx added in v0.3.2 

func (r *Repo) DownloadFilesCtx(ctx context.Context, repoFiles ...string) (downloadedPaths []string, err error)

DownloadFilesCtx is like DownloadFiles but accepts a context for cancellation support.

func (*Repo) DownloadInfo 

func (r *Repo) DownloadInfo(forceDownload bool) error

DownloadInfo about the model, if it hasn't yet.

It will attempt to use the "_info_.json" file in the cache directory first.

If forceDownload is set to true, it ignores the current info or the cached one, and download it again from HuggingFace.

See Repo.Info to access the Info directory. Most users don't need to call this directly, instead use the various iterators.

func (*Repo) FileURL 

func (r *Repo) FileURL(fileName string) (string, error)

FileURL returns the URL from which to download the file from HuggingFace.

Usually, not used directly (use DownloadFile instead), but in case someone needs for debugging.

func (*Repo) GetDownloadManager added in v0.3.5 

func (r *Repo) GetDownloadManager() *downloader.Manager

GetDownloadManager returns current downloader.Manager, or creates a new one for this Repo.

Internal use only.

func (*Repo) HasFile 

func (r *Repo) HasFile(fileName string) bool

HasFile returns whether the repo has given fileName. Notice fileName is relative to the repository, not in local disk.

If the Repo hasn't downloaded its info yet, it attempts to download it here. If it fails, it simply return false. Call Repo.DownloadInfo to handle errors downloading the info.

func (*Repo) Info 

func (r *Repo) Info() *RepoInfo

Info returns the RepoInfo structure about the model. Most users don't need to call this directly, instead use the various iterators.

If it hasn't been downloaded or loaded from the cache yet, it loads it first.

It may return nil if there was an issue with the downloading of the RepoInfo json from HuggingFace. Try DownloadInfo to get an error.

func (*Repo) IterFileInfos added in v0.4.0 

func (r *Repo) IterFileInfos() iter.Seq2[*FileInfo, error]

IterFileInfos iterate over the FileInfo of the files stored in the repo. It doesn't trigger the downloading of the repo, only of the repo info.

func (*Repo) IterFileNames 

func (r *Repo) IterFileNames() iter.Seq2[string, error]

IterFileNames iterate over the file names stored in the repo. It doesn't trigger the downloading of the repo, only of the repo info.

func (*Repo) String 

func (r *Repo) String() string

String implements fmt.Stringer.

func (*Repo) WithAuth 

func (r *Repo) WithAuth(authToken string) *Repo

WithAuth sets the authentication token to use during downloads.

Setting it to empty ("") is the same as resetting and not using authentication.

func (*Repo) WithCacheDir 

func (r *Repo) WithCacheDir(cacheDir string) *Repo

WithCacheDir sets the cacheDir to the given directory.

The default is given by DefaultCacheDir: `${XDG_CACHE_HOME}/huggingface/hub` if set, or `~/.cache/huggingface/hub` otherwise.

func (*Repo) WithDownloadManager 

func (r *Repo) WithDownloadManager(manager *downloader.Manager) *Repo

WithDownloadManager sets the downloader.Manager to use for download. This is not needed, one will be created automatically if one is not set. This is useful when downloading multiple Repos simultaneously, to coordinate limits by sharing the download manager.

func (*Repo) WithEndpoint added in v0.1.2 

func (r *Repo) WithEndpoint(endpoint string) *Repo

WithEndpoint sets the HuggingFace endpoint to use. Default is "https://huggingface.co" or, if set, the environment variable HF_ENDPOINT.

func (*Repo) WithExtraBlobsInfo added in v0.4.0 

func (r *Repo) WithExtraBlobsInfo(extraBlobsInfo bool) *Repo

WithExtraBlobsInfo sets whether to retrieve the underlying Git and LFS metadata for the repository's files. This includes file sizes.

Default true. Set this to false to save some bandwidth if only donwloading the info.

func (*Repo) WithProgressBar 

func (r *Repo) WithProgressBar(useProgressBar bool) *Repo

WithProgressBar configures the usage of progress bar during download. Defaults to true.

func (*Repo) WithRevision 

func (r *Repo) WithRevision(revision string) *Repo

WithRevision sets the revision to use for this Repo, defaults to "main", but can be set to a commit-hash value.

func (*Repo) WithType 

func (r *Repo) WithType(repoType RepoType) *Repo

WithType sets the repository type to use during downloads.

type RepoConfig added in v0.4.0 

type RepoConfig struct {
	Architectures   []string       `json:"architectures"`
	ModelType       string         `json:"model_type"`
	TokenizerConfig map[string]any `json:"tokenizer_config"`
}

RepoConfig contains configuration info about the repository/model.

type RepoInfo 

type RepoInfo struct {
	InternalID   string            `json:"_id"`
	ID           string            `json:"id"`
	ModelID      string            `json:"modelId"` // Also support model_id via custom UnmarshalJSON
	Author       string            `json:"author"`
	CommitHash   string            `json:"sha"`
	PipelineTag  string            `json:"pipeline_tag"`
	Tags         []string          `json:"tags"`
	Siblings     []*FileInfo       `json:"siblings"`
	SafeTensors  SafeTensorsInfo   `json:"safetensors"`
	Private      bool              `json:"private"`
	LibraryName  string            `json:"library_name"`
	Downloads    int64             `json:"downloads"`
	Likes        int64             `json:"likes"`
	LastModified time.Time         `json:"lastModified"`
	Gated        any               `json:"gated"`
	Disabled     bool              `json:"disabled"`
	WidgetData   []map[string]any  `json:"widgetData"`
	WidgetInfo   map[string]any    `json:"widgetInfo"`
	ModelIndex   any               `json:"model-index"`
	Config       *RepoConfig       `json:"config"`
	CardData     *CardData         `json:"cardData"`
	Transformers *TransformersInfo `json:"transformersInfo"`
	Spaces       []string          `json:"spaces"`
	CreatedAt    time.Time         `json:"createdAt"`
	UsedStorage  int64             `json:"usedStorage"`
}

RepoInfo holds information about a HuggingFace repo, it is the json served when hitting the URL https://huggingface.co/api/<repo_type>/<model_id>?blobs=true

func (*RepoInfo) UnmarshalJSON added in v0.4.0 

func (r *RepoInfo) UnmarshalJSON(data []byte) error

UnmarshalJSON customizes unmarshaling for RepoInfo to support both "modelId" and "model_id".

type RepoType 

type RepoType string

RepoType supported by HuggingFace-Hub 

const (
	RepoTypeDataset RepoType = "datasets"
	RepoTypeSpace   RepoType = "spaces"
	RepoTypeModel   RepoType = "models"
)

type SafeTensorsInfo 

type SafeTensorsInfo struct {
	Total      int64            `json:"total"`
	Parameters map[string]int64 `json:"parameters"`
}

SafeTensorsInfo holds counts on number of parameters of various types.

type TransformersInfo added in v0.4.0 

type TransformersInfo struct {
	AutoModel   string `json:"auto_model"`
	PipelineTag string `json:"pipeline_tag"`
	Processor   string `json:"processor"`
}

TransformersInfo contains information related to the Hugging Face Transformers integration.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.