githosts

package module
v0.0.0-...-24e4118 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 10, 2026 License: MIT Imports: 27 Imported by: 1

README

githosts-utils

githosts-utils is a Go library for backing up repositories from major hosting providers. It powers soba and can be embedded in your own tools.

Features

  • Minimal dependencies and portable code
  • Supports GitHub, GitLab, Bitbucket, Azure DevOps, Gitea, and Sourcehut
  • Clones repositories using git --mirror and stores timestamped bundle files
  • Encryption support: Optional age-based encryption for bundles and manifests
  • Optional reference comparison to skip cloning when refs have not changed
  • Ability to keep a configurable number of previous bundles
  • Optional Git LFS archival alongside each bundle
  • Pluggable HTTP client and simple logging via the GITHOSTS_LOG environment variable

Installation

go get github.com/jonhadfield/githosts-utils

Requires Go 1.22 or later.

Quick Start

Create a host for the provider you want to back up and call Backup() on it. Each provider has its own input struct with the required options. The example below backs up a set of GitHub repositories:

package main

import (
    "log"
    "os"

    "github.com/jonhadfield/githosts-utils"
)

func main() {
    backupDir := "/path/to/backups"

    host, err := githosts.NewGitHubHost(githosts.NewGitHubHostInput{
        Caller:               "example",
        BackupDir:            backupDir,
        Token:                os.Getenv("GITHUB_TOKEN"),
        BackupLFS:            true,
        EncryptionPassphrase: os.Getenv("BUNDLE_PASSPHRASE"), // Optional encryption
    })
    if err != nil {
        log.Fatal(err)
    }

    results := host.Backup()
    for _, r := range results.BackupResults {
        log.Printf("%s: %s", r.Repo, r.Status)
    }
}

Backup() returns a ProviderBackupResult containing the status of each repository. Bundles are written beneath <backupDir>/<provider>/<owner>/<repo>/.

Diff Remote Method

Each host accepts a DiffRemoteMethod value of either "clone" or "refs":

  • clone (default) – always clone and create a new bundle
  • refs – fetch remote references first and skip cloning when the refs match the latest bundle
Retaining Bundles

Set BackupsToRetain to keep only the most recent n bundle files per repository. Older bundles are automatically deleted after a successful backup.

Encryption

The library supports optional age-based encryption for all backup bundles and their associated manifests. When encryption is enabled:

  • Bundle files are encrypted and saved with a .age extension (e.g., repo.20240101000000.bundle.age)
  • Manifest files containing bundle metadata and git refs are also encrypted
  • The system can seamlessly work with both encrypted and unencrypted bundles in the same repository
Enabling Encryption

You can enable encryption in two ways:

  1. Via environment variable: Set BUNDLE_PASSPHRASE to your encryption passphrase
  2. Via host configuration: Pass the EncryptionPassphrase parameter when creating a host
host, err := githosts.NewGitHubHost(githosts.NewGitHubHostInput{
    BackupDir:            backupDir,
    Token:                token,
    EncryptionPassphrase: "your-secure-passphrase",
})
Encryption Behavior
  • When using the refs diff method, the system can compare encrypted bundles without decrypting them by using manifest files
  • If you switch from encrypted to unencrypted backups (or vice versa), the system handles this gracefully
  • Wrong passphrases are detected and reported with appropriate error messages
  • Corrupted encrypted files are handled safely with fallback mechanisms

Environment Variables

The library reads the following variables where relevant:

  • GITHOSTS_LOG – set to debug to emit verbose logs
  • GIT_BACKUP_DIR – used by the tests to determine the backup location
  • BUNDLE_PASSPHRASE – optional passphrase for encrypting backup bundles

Provider-specific tests require credentials through environment variables such as GITHUB_TOKEN, GITLAB_TOKEN, BITBUCKET_KEY, BITBUCKET_SECRET, AZURE_DEVOPS_USERNAME, AZURE_DEVOPS_PAT, GITEA_TOKEN, and SOURCEHUT_TOKEN.

Running Tests

export GIT_BACKUP_DIR=$(mktemp -d)
go test ./...

Integration tests are skipped unless the corresponding provider credentials are present.

License

This project is licensed under the MIT License. See LICENSE for details.

Documentation

Overview

getEnvOrFile returns the value of the environment variable if set, otherwise if a corresponding _FILE variable is set, reads the value from the file at that path. If both are set, the environment variable takes precedence.

Index

Constants

View Source 
const (
	BitbucketProviderName = "BitBucket"

	// Auth Type
	AuthTypeBitbucketOAuth2   = AuthTypeBearerToken
	AuthTypeBitbucketAPIToken = AuthTypeBasicAuthHeader
	AuthTypeBasicAuthHeader   = "basic-auth-header"
	AuthTypeBearerToken       = "bearer-token"
)
View Source 
const (

	// HTTP Headers
	HeaderContentType   = "Content-Type"
	HeaderAuthorization = "Authorization"
	HeaderAccept        = "Accept"

	// Authentication prefixes
	AuthPrefixBearer = "Bearer "
	AuthPrefixToken  = "token "
	AuthPrefixBasic  = "Basic "

	// Content types
	ContentTypeJSON        = "application/json"
	ContentTypeFormEncoded = "application/x-www-form-urlencoded"
	ContentTypeAny         = "*/*"
)
View Source 
const (
	AzureDevOpsProviderName = "AzureDevOps"
)
View Source 
const (
	// GitLabDefaultMinimumProjectAccessLevel https://docs.gitlab.com/ee/user/permissions.html#roles
	GitLabDefaultMinimumProjectAccessLevel = 20
)

Variables

This section is empty.

Functions

func AddBasicAuthToURL 

func AddBasicAuthToURL(originalURL, username, password string) (string, error)

func ToPtr 

func ToPtr[T any](v T) *T

func TrimInPlace 

func TrimInPlace(s *string)

Types

type AzureDevOpsHost 

type AzureDevOpsHost struct {
	Caller               string
	HttpClient           *retryablehttp.Client
	Provider             string
	PAT                  string
	Orgs                 []string
	UserName             string
	DiffRemoteMethod     string
	BackupDir            string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

func NewAzureDevOpsHost 

func NewAzureDevOpsHost(input NewAzureDevOpsHostInput) (*AzureDevOpsHost, error)

func (*AzureDevOpsHost) Backup 

func (ad *AzureDevOpsHost) Backup() ProviderBackupResult

type AzureDevOpsRepo 

type AzureDevOpsRepo struct {
	Id            string  `json:"id"`
	Url           string  `json:"url"`
	Name          string  `json:"name"`
	Size          int64   `json:"size"`
	SshUrl        string  `json:"sshUrl"`
	WebUrl        string  `json:"webUrl"`
	Project       Project `json:"project"`
	RemoteUrl     string  `json:"remoteUrl"`
	DefaultBranch string  `json:"defaultBranch"`
}

func ListAllRepositories 

func ListAllRepositories(httpClient *retryablehttp.Client, basicAuth, projectName, orgName string) ([]AzureDevOpsRepo, error)

type BasicAuth 

type BasicAuth struct {
	User     string `json:"user,omitempty"`
	Password string `json:"password,omitempty"`
}

type BitbucketHost 

type BitbucketHost struct {
	Caller           string
	HttpClient       *retryablehttp.Client
	Provider         string
	APIURL           string
	DiffRemoteMethod string
	BackupDir        string
	BackupsToRetain  int
	AuthType         string
	// API OAuthToken
	Email     string
	APIToken  string
	BasicAuth BasicAuth
	// OAuth2
	User                 string
	OAuthToken           string
	Key                  string
	Secret               string
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

func NewBitBucketHost 

func NewBitBucketHost(input NewBitBucketHostInput) (*BitbucketHost, error)

func (BitbucketHost) Backup 

func (bb BitbucketHost) Backup() ProviderBackupResult

type BundleManifest 

type BundleManifest struct {
	CreationTime string            `json:"creation_time"`
	BundleHash   string            `json:"bundle_hash"`
	BundleFile   string            `json:"bundle_file"`
	GitRefs      map[string]string `json:"git_refs"`
}

BundleManifest represents the metadata for a bundle

type GitHubHost 

type GitHubHost struct {
	Caller               string
	HttpClient           *retryablehttp.Client
	Provider             string
	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	SkipUserRepos        bool
	LimitUserOwned       bool
	BackupsToRetain      int
	Token                string
	Orgs                 []string
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

func NewGitHubHost 

func NewGitHubHost(input NewGitHubHostInput) (*GitHubHost, error)

func (*GitHubHost) Backup 

func (gh *GitHubHost) Backup() ProviderBackupResult

type GitLabHost 

type GitLabHost struct {
	Caller string

	APIURL                string
	DiffRemoteMethod      string
	BackupDir             string
	BackupsToRetain       int
	ProjectMinAccessLevel int
	Token                 string
	User                  gitlabUser
	LogLevel              int
	BackupLFS             bool
	EncryptionPassphrase  string
	// contains filtered or unexported fields
}

func NewGitLabHost 

func NewGitLabHost(input NewGitLabHostInput) (*GitLabHost, error)

func (*GitLabHost) Backup 

func (gl *GitLabHost) Backup() ProviderBackupResult

type GiteaHost 

type GiteaHost struct {
	Caller string

	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	BackupsToRetain      int
	Token                string
	Orgs                 []string
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
	// contains filtered or unexported fields
}

func NewGiteaHost 

func NewGiteaHost(input NewGiteaHostInput) (*GiteaHost, error)

func (*GiteaHost) Backup 

func (g *GiteaHost) Backup() ProviderBackupResult

type LFSManifest 

type LFSManifest struct {
	CreationTime string `json:"creation_time"`
	ArchiveHash  string `json:"archive_hash"`
	ArchiveFile  string `json:"archive_file"`
}

LFSManifest represents the metadata for an LFS archive

type NewAzureDevOpsHostInput 

type NewAzureDevOpsHostInput struct {
	HTTPClient           *retryablehttp.Client
	Caller               string
	BackupDir            string
	DiffRemoteMethod     string
	UserName             string
	PAT                  string
	Orgs                 []string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

type NewBitBucketHostInput 

type NewBitBucketHostInput struct {
	Caller           string
	HTTPClient       *retryablehttp.Client
	APIURL           string
	DiffRemoteMethod string
	BackupDir        string
	// API OAuthToken
	Email     string
	BasicAuth BasicAuth
	AuthType  string
	// API OAuthToken
	APIToken string
	// OAuth2
	User                 string
	Key                  string
	Secret               string
	OAuthToken           string
	Username             string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

type NewGitHubHostInput 

type NewGitHubHostInput struct {
	HTTPClient           *retryablehttp.Client
	Caller               string
	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	Token                string
	LimitUserOwned       bool
	SkipUserRepos        bool
	Orgs                 []string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

type NewGitLabHostInput 

type NewGitLabHostInput struct {
	Caller                string
	HTTPClient            *retryablehttp.Client
	APIURL                string
	DiffRemoteMethod      string
	BackupDir             string
	Token                 string
	ProjectMinAccessLevel int
	BackupsToRetain       int
	LogLevel              int
	BackupLFS             bool
	EncryptionPassphrase  string
}

type NewGiteaHostInput 

type NewGiteaHostInput struct {
	Caller               string
	HTTPClient           *retryablehttp.Client
	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	Token                string
	Orgs                 []string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

type NewSourcehutHostInput 

type NewSourcehutHostInput struct {
	HTTPClient           *retryablehttp.Client
	Caller               string
	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	PersonalAccessToken  string
	LimitUserOwned       bool
	SkipUserRepos        bool
	Orgs                 []string
	BackupsToRetain      int
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

type Project 

type Project struct {
	Id             string    `json:"id"`
	Url            string    `json:"url"`
	Name           string    `json:"name"`
	State          string    `json:"state"`
	Revision       int       `json:"revision"`
	Visibility     string    `json:"visibility"`
	Description    string    `json:"description"`
	LastUpdateTime time.Time `json:"lastUpdateTime"`
}

type ProviderBackupResult 

type ProviderBackupResult struct {
	BackupResults []RepoBackupResults
	Error         errors.E
}

type RepoBackupResults 

type RepoBackupResults struct {
	Repo   string   `json:"repo,omitempty"`
	Status string   `json:"status,omitempty"` // ok, failed
	Error  errors.E `json:"error,omitempty"`
}

type SourcehutHost 

type SourcehutHost struct {
	Caller               string
	HttpClient           *retryablehttp.Client
	Provider             string
	APIURL               string
	DiffRemoteMethod     string
	BackupDir            string
	SkipUserRepos        bool
	LimitUserOwned       bool
	BackupsToRetain      int
	PersonalAccessToken  string
	Orgs                 []string
	LogLevel             int
	BackupLFS            bool
	EncryptionPassphrase string
}

func NewSourcehutHost 

func NewSourcehutHost(input NewSourcehutHostInput) (*SourcehutHost, error)

func (*SourcehutHost) Backup 

func (sh *SourcehutHost) Backup() ProviderBackupResult

type WorkerConfig 

type WorkerConfig struct {
	LogLevel             int
	BackupDir            string
	DiffRemoteMethod     string
	BackupsToKeep        int
	BackupLFS            bool
	DefaultDelay         int
	DelayEnvVar          string
	Secrets              []string
	SetupRepo            func(*repository) // Function to set up authentication on the repo
	EncryptionPassphrase string
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.