backup

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2026 License: GPL-3.0 Imports: 20 Imported by: 0

README

backup-git-repos

Test Lint Prose Go Reference Release Licence

Backs up every repository on a self-hosted GitLab or Forgejo instance to a local, restorable copy.

A backup only counts if it survives the forge going away, so what backup-git-repos keeps is a bare mirror clone of each repository: every branch, every tag, and the same namespace folder structure the forge used. It tells archived repositories from active ones, and can back up either set, both, or write either out as a .tar.gz alongside the mirror.

Features

  • Mirrors every branch, tag and ref, not just the default branch
  • Keeps the forge's own namespace structure on disk
  • Filters by archived, active, or all repositories
  • Refreshes existing mirrors incrementally instead of re-cloning
  • Optionally writes archived, active, or all repositories out as .tar.gz
  • Backs up several forges in one run: GitLab instances, Forgejo instances, or a mix of both

Requirements

  • Go 1.26 or newer to build it.
  • git, on PATH, to do the actual cloning. The tool shells out to it rather than re-implementing the protocol, which is what makes an incremental mirror refresh fast and the resulting .git directory exactly what git clone produces anywhere else.
  • A personal access token for each forge you back up, with read access to every repository you want. See Configuration for where it goes — never in the config file itself.
  • Somewhere to write the backup tree. It grows to roughly the size of every repository you're backing up, twice over if you also enable .tar.gz archives.

Working on the tool needs more than running it does; that list is in CONTRIBUTING.md.

Installation

go install github.com/alrayyes/backup-git-repos/cmd/backup-git-repos@latest

Or build from a clone:

git clone https://github.com/alrayyes/backup-git-repos.git
cd backup-git-repos
go build -o backup-git-repos ./cmd/backup-git-repos

Released binaries are attached to each GitHub release.

Configuration

List every forge in a YAML file. A token never goes in the file, only the name of the environment variable holding it — the tool reads the variable at startup and fails before touching the network if it's unset.

dest: /srv/backups/git
forges:
  - name: work # becomes the top-level folder for this forge's repos
    kind: gitlab
    url: https://gitlab.example.com
    token_env: WORK_GITLAB_TOKEN
  - name: home
    kind: forgejo
    url: https://git.example.org
    token_env: HOME_FORGEJO_TOKEN

export WORK_GITLAB_TOKEN=glpat-...
export HOME_FORGEJO_TOKEN=...

Usage

backup-git-repos run --config ./config.yaml

Resulting layout:

/srv/backups/git/
  work/group/subgroup/repo.git/           # bare mirror
  home/team/repo.git/
  archive/work/group/subgroup/repo.tar.gz # only when --archive is set
Restoring a repository

A mirror is a normal bare repository, so cloning out of it is the whole restore:

git clone /srv/backups/git/home/team/repo.git restored-repo

From an archive, extract first:

tar xzf /srv/backups/git/archive/home/team/repo.tar.gz
git clone repo.git restored-repo
Flags
  • --config, -c: path to the YAML config (required)
  • --dest, -d: override the destination directory from the config. Required on run unless the config sets dest
  • --forge: repeatable; restrict the run to named forges
  • --state: all | active | archived — which repositories to mirror (default all)
  • --archive: none | all | active | archived — which repositories also get written out as a .tar.gz (default none)
  • --archive-dir: where archives go (default <dest>/archive)
  • --concurrency, -j: repositories mirrored in parallel (default the number of CPUs)
  • --timeout: per-repository timeout (default 30m)

backup-git-repos list runs the same discovery and filtering without cloning anything, which is the fast way to check a config is picking up what you expect.

License

This project is licensed under the GNU General Public License v3.0 — see the LICENSE file for details.

Contributing

Contributions are welcome. Open a pull request.

Read CONTRIBUTING.md first. It covers the setup, how the test suites are split, and what each linter is for.

Documentation

Overview

Package backup mirrors git repositories out of self-hosted forges.

A backup has to survive the forge going away, so what it keeps is a bare mirror clone of every repository: every branch, every tag, and the namespace folder structure the forge used, refreshed in place on later runs. Archived and active repositories can be selected separately, and either set can be written out as a tar.gz alongside the mirror.

Index

Constants

View Source 
const (
	TestActiveRepoPath   = "team/active-repo"
	TestArchivedRepoPath = "team/archived-repo"
	TestEmptyRepoPath    = "team/empty-repo"
)

The paths every Lister's and Runner's fixture data is expected to seed. An adapter's own fixture-seeding driver (the fake, Forgejo, GitLab) is responsible for creating repositories under these exact paths, which is what lets the suites below run unchanged against any of them.

Variables

View Source 
var ErrBadArchive = errors.New("archive must be one of: none, all, active, archived")

ErrBadArchive means --archive wasn't one of none, all, active, or archived.

View Source 
var ErrBadState = errors.New("state must be one of: all, active, archived")

ErrBadState means --state wasn't one of all, active, or archived.

View Source 
var ErrGitNotFound = errors.New("git not found on PATH")

ErrGitNotFound means git isn't on PATH.

View Source 
var ErrMissingToken = errors.New("token environment variable not set")

ErrMissingToken means a forge's token_env names an environment variable that isn't set.

Functions

func Archive 

func Archive(dir, out string) error

Archive writes a gzipped tar of the mirror at dir to out, atomically: it writes to out+".tmp" first and renames over out only on success, so a crash mid-write never leaves a truncated archive where a good one was.

Every entry is written under a top-level directory named after dir's own base name, so "tar xzf out" leaves a self-contained "<name>.git/" a caller can git clone directly rather than scattering the mirror's contents into whatever directory they happened to extract into.

Entries are read through an os.Root rooted at dir, which is what stops a symlink inside the mirror from making the walk reference anything outside it. Every entry's ModTime, Uid, Gid, Uname and Gname are zeroed, and entries are written in sorted path order, so archiving an unchanged mirror twice produces a byte-identical file -- which is what makes rsync or deduplication of the backup tree cheap.

func NewRootCommand 

func NewRootCommand(version string, newRunner NewRunner) *cobra.Command

NewRootCommand builds the backup-git-repos command line. Each RunE stays a thin shell: parse and validate the flags, then call into runBackup, which knows nothing about cobra.

func TestBackup 

func TestBackup(t *testing.T, run TestDriver)

TestBackup runs the specification every forge's backup pipeline must satisfy, in domain terms: it talks only about a destination directory and the state it asks for, so the same spec runs unchanged against a fake or a real forge.

func TestLister 

func TestLister(t *testing.T, newLister func(t *testing.T) Lister)

TestLister runs the behaviour every Lister must satisfy, whether it's backed by a fake or a real forge. Exported here, rather than living in a _test.go file, so an adapter under internal/ can run the same suite against itself -- otherwise an unexported helper in this package's own tests would be unreachable from anywhere that isn't this package.

Types

type ArchiveSelection 

type ArchiveSelection int

ArchiveSelection selects which repositories also get written out as a tar.gz alongside their mirror, in addition to being mirrored. 

const (
	ArchiveNone ArchiveSelection = iota
	ArchiveAll
	ArchiveActive
	ArchiveArchived
)

The values ArchiveSelection can hold.

func ParseArchive 

func ParseArchive(s string) (ArchiveSelection, error)

ParseArchive parses a --archive flag value.

type Config 

type Config struct {
	Dest   string        `yaml:"dest"`
	Forges []ForgeConfig `yaml:"forges"`
}

Config is the top-level shape of the YAML config file.

func LoadConfig 

func LoadConfig(path string) (Config, error)

LoadConfig reads and validates the config file at path: every forge's kind must be one this build supports, and every forge's token_env must name a set environment variable. Failing before touching the network beats a 401 partway through a run.

type ForgeConfig 

type ForgeConfig struct {
	Name     string `yaml:"name"`
	Kind     string `yaml:"kind"`
	URL      string `yaml:"url"`
	TokenEnv string `yaml:"token_env"`
	Token    string `yaml:"-"`
}

ForgeConfig is one forge entry from the config file. Token is resolved from the environment variable named by TokenEnv during LoadConfig, never read from the file itself.

type Lister 

type Lister interface {
	ListRepos(ctx context.Context, state State) ([]Repo, error)
}

Lister lists the repositories a forge holds, filtered by state.

type Mirror 

type Mirror struct {
	GitPath string
}

Mirror keeps a bare mirror clone of a repository up to date on disk. The zero value works: it resolves git from PATH.

func (Mirror) Sync 

func (m Mirror) Sync(ctx context.Context, r Remote, dir string) error

Sync creates a bare mirror at dir if it doesn't exist yet, or refreshes an existing one otherwise. A refresh prunes refs the remote no longer has.

type Mirrorer 

type Mirrorer interface {
	Sync(ctx context.Context, r Remote, dir string) error
}

Mirrorer keeps a repository's bare mirror up to date on disk.

type NewRunner 

type NewRunner func(ForgeConfig) (Runner, error)

NewRunner builds the Runner for a configured forge. Implementations live with their adapters under internal/, so this package -- which every adapter imports for Repo, Lister and the rest -- can't reference them directly without a cycle. The composition root (main) supplies one.

type Options 

type Options struct {
	Dest        string
	State       State
	Archive     ArchiveSelection
	ArchiveDir  string
	Concurrency int
	Timeout     time.Duration
	Log         *slog.Logger
}

Options configures a Run.

type Remote 

type Remote struct {
	CloneURL   string
	AuthHeader string
}

Remote is a repository's clone URL and, where the forge needs one, the value of the Authorization header to send with it. Keeping the credential in a header rather than the URL is what keeps it out of the mirror's own .git/config -- a URL embedded there would leave the token in the clear in every mirror on disk.

type Remoter 

type Remoter interface {
	Remote(repo Repo) Remote
}

Remoter builds the clone Remote for a repository.

type Repo 

type Repo struct {
	Path     string
	Archived bool
	Empty    bool
}

Repo is a repository as reported by a forge: where it lives in the forge's namespace, and whether it's archived or carries no commits yet.

type Result 

type Result struct {
	Synced   int
	Skipped  int
	Failed   int
	Archived int
}

Result reports what a Run did.

type Runner 

type Runner struct {
	Lister   Lister
	Mirrorer Mirrorer
	Remoter  Remoter
}

Runner backs up one forge: it lists repositories, then mirrors each one into a destination tree that mirrors the forge's own namespace structure.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, opts Options) (Result, error)

Run lists repositories filtered by opts.State and mirrors each one, skipping repositories with no refs -- a mirror of one would just confuse the next refresh. It stays synchronous even though it mirrors up to opts.Concurrency repositories at once: the call blocks until every repo has been attempted, and the caller decides whether to run it concurrently with anything else. One repository failing is logged and doesn't stop the rest; Result.Failed is how the caller finds out afterwards.

type State 

type State int

State selects which repositories a Lister returns. 

const (
	StateAll State = iota
	StateActive
	StateArchived
)

The states a Lister can be asked to filter on.

func ParseState 

func ParseState(s string) (State, error)

ParseState parses a --state flag value.

func (State) String 

func (s State) String() string

type TestDriver 

type TestDriver func(ctx context.Context, opts Options) (Result, error)

TestDriver runs a backup the way Runner.Run does, whether that's against an in-memory fake or a real forge container.

type UnknownKindError 

type UnknownKindError struct {
	Kind string
}

UnknownKindError means a forge's kind isn't one backup-git-repos knows how to talk to.

func (*UnknownKindError) Error 

func (e *UnknownKindError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
backup-git-repos command
Command backup-git-repos mirrors git repositories out of self-hosted forges.
 Command backup-git-repos mirrors git repositories out of self-hosted forges.
internal
forgejo
Package forgejo lists and mirrors repositories from a self-hosted Forgejo (or Gitea) instance over its REST API.
 Package forgejo lists and mirrors repositories from a self-hosted Forgejo (or Gitea) instance over its REST API.
gitlab
Package gitlab lists and mirrors repositories from a self-hosted GitLab instance over its REST API.
 Package gitlab lists and mirrors repositories from a self-hosted GitLab instance over its REST API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.