backup

package module
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: GPL-3.0 Imports: 24 Imported by: 0

README

backup-git-repos

Test Codecov Lint Prose Go Reference Release Licence

Backs up every repository on a self-hosted GitLab or Forgejo instance, or a GitHub.com account, to a local, restorable copy.

A backup only counts if it survives the forge going away, so what backup-git-repos keeps is a bare mirror clone of each repository: every branch, every tag, and the same namespace folder structure the forge used. It tells archived repositories from active ones, and can back up either set, both, or write either out as a .tar.gz alongside the mirror.

Features

  • Mirrors every branch, tag and ref, not just the default branch
  • Keeps the forge's own namespace structure on disk
  • Filters by archived, active, or all repositories
  • Refreshes existing mirrors incrementally instead of re-cloning
  • Optionally writes archived, active, or all repositories out as .tar.gz, a gzipped copy of the bare mirror itself -- not a working-tree checkout, so every branch and tag survives the archive too
  • Backs up several forges in one run: GitLab instances, Forgejo instances, GitHub.com accounts, or a mix

Requirements

  • Go 1.26 or newer to build it.
  • git, on PATH, to do the actual cloning. The tool shells out to it rather than re-implementing the protocol, which is what makes an incremental mirror refresh fast and the resulting .git directory exactly what git clone produces anywhere else.
  • A personal access token for each forge you back up, with read access to every repository you want. See Configuration for where it goes.
  • Somewhere to write the backup tree. It grows to roughly the size of every repository you're backing up, twice over if you also enable .tar.gz archives.

Working on the tool needs more than running it does; that list is in CONTRIBUTING.md.

Installation

go install github.com/alrayyes/backup-git-repos/cmd/backup-git-repos@latest

Or build from a clone:

git clone https://github.com/alrayyes/backup-git-repos.git
cd backup-git-repos
go build -o backup-git-repos ./cmd/backup-git-repos

Released binaries are attached to each GitHub release.

Or run the container image, git and all — a natural fit for a scheduled job. It runs as UID/GID 1000, not root, so the destination directory needs to be writable by that UID on the host:

mkdir -p /srv/backups/git && chown 1000:1000 /srv/backups/git
docker run --rm \
  --read-only --tmpfs /tmp \
  --cap-drop=ALL --security-opt=no-new-privileges \
  --memory=512m --cpus=2 \
  -v /srv/backups/git:/srv/backups/git -v ./config.yaml:/config.yaml:ro \
  ghcr.io/alrayyes/backup-git-repos:latest run --config /config.yaml

The image needs no capability beyond the default network access every container gets, so --cap-drop=ALL alone is the whole capability line, with nothing to add back. --tmpfs /tmp matters specifically for --archive: an already-archived repository mirrors into a scratch directory under /tmp before being written out as a .tar.gz (see Usage), which needs somewhere writable once --read-only locks the rest of the image's own filesystem. Size the mount for your largest archived repository if the default (half the host's RAM) isn't enough, and size --memory/--cpus for how many repositories --concurrency mirrors at once.

Images are multi-arch (linux/amd64, linux/arm64), tagged latest and per version, and published alongside every release at ghcr.io/alrayyes/backup-git-repos.

Configuration

List every forge in a YAML file, the token included. That's the default and recommended form: paste the token straight into token, and there's nothing else to wire up before run works. Treat a config carrying one as a secret, the same care you'd give any file with credentials in it — chmod 600 it, and don't check it in. If you'd rather keep the token out of the file entirely, token_env is the opt-in for that; see below.

backup-git-repos config init writes a starter file to get from a blank directory to something you can edit — $XDG_CONFIG_HOME/backup-git-repos/config.yaml (or ~/.config/backup-git-repos/config.yaml if XDG_CONFIG_HOME isn't set) by default, or wherever --config/-c names. It refuses to overwrite an existing file unless you pass --force.

run and list read from that same default path when you don't pass --config yourself, so once config init has written it, plain backup-git-repos run is enough. --config still wins when you pass it, and without it and with nothing at the default path either, the tool exits telling you which path it checked.

backup-git-repos config init

dest: /srv/backups/git
forges:
  - name: work # becomes the top-level folder for this forge's repos
    kind: gitlab
    url: https://gitlab.example.com
    token: glpat-...
  - name: home
    kind: forgejo
    url: https://git.example.org
    token: ...
  - name: personal
    kind: github
    token: ghp_...

url is only for a self-hosted forge; GitHub.com is the one instance, so a github entry never sets it.

Each kind's token needs enough to list and clone every repository you want backed up, and nothing more:

  • GitLab: a personal access token with the read_api and read_repository scopes. read_api covers listing projects; read_repository covers the git clone itself.
  • Forgejo: a token with the read:repository scope (fine-grained tokens) or, on a version old enough to only offer full-access tokens, one scoped as narrowly as your Forgejo instance allows.
  • GitHub: the classic repo scope, or a fine-grained token with read access to contents and metadata on every repository you want backed up. For a classic token, run and list check the scopes GitHub reports back and fail fast if repo is missing, rather than silently backing up only the public repositories. GitHub gives no equivalent signal for a fine-grained token, so that check can't run against one -- double-check a fine-grained token's repository access is set to what you expect.

A forgejo entry can also set skip_mirrors: true to exclude repositories Forgejo itself reports as mirrors of an external upstream from both listing and mirroring — there's no point re-backing-up content that already lives at its real source elsewhere. Off by default; other kinds ignore the field.

A forge entry can set token_env instead of token: the name of an environment variable holding the token, if you'd rather keep it out of the file — the tool reads the variable at startup and fails before touching the network if it's unset. Setting both on the same entry is a config error, not a silent pick-one. Reach for it when whatever backs up this tool's own backup tree (a git remote, a sync job, a snapshot) would otherwise pick up a token in the file the same as any other line in it:

forges:
  - name: work
    kind: gitlab
    url: https://gitlab.example.com
    token_env: WORK_GITLAB_TOKEN

export WORK_GITLAB_TOKEN=glpat-...

Usage

backup-git-repos run --config ./config.yaml

Resulting layout:

/srv/backups/git/
  work/group/subgroup/repo.git/           # bare mirror, active repo
  home/team/repo.git/
  archive/work/group/subgroup/repo.tar.gz # only when --archive is set
  archive/home/team/old-repo.tar.gz       # archived repo selected by --archive:
                                           # only the tar.gz, no .git alongside it
Restoring a repository

A .tar.gz holds the same bare mirror as the .git directory next to it, not a working-tree checkout -- extracting it alone doesn't hand you files you can edit. That's deliberate: a bare mirror is the only form that keeps every branch and tag, not just whichever one would've been checked out. git clone is what turns either one into an ordinary working copy.

A mirror is a normal bare repository, so cloning out of it is the whole restore:

git clone /srv/backups/git/home/team/repo.git restored-repo

From an archive, extract first:

tar xzf /srv/backups/git/archive/home/team/repo.tar.gz
git clone repo.git restored-repo
Flags
  • --config, -c: path to the YAML config (default: $XDG_CONFIG_HOME/backup-git-repos/config.yaml, falling back to ~/.config/backup-git-repos/config.yaml; required if no file exists at that path)
  • --dest, -d: override the destination directory from the config. Required on run unless the config sets dest. A leading ~ (in this flag, in dest in the config file, or in --archive-dir) expands to your home directory
  • --forge: repeatable; restrict the run to named forges
  • --state: all | active | archived — which repositories to mirror (default all)
  • --archive: none | all | active | archived — which repositories also get written out as a .tar.gz (default none)
  • --archive-dir: where archives go (default <dest>/archive)
  • --concurrency, -j: repositories mirrored in parallel (default the number of CPUs)
  • --timeout: per-repository timeout (default 30m)
  • --verbose, -v: log each repository as it starts mirroring and archiving, not just failures and the final summary
  • --dry-run: print what the run would do -- clone or update per repository, plus archive where --archive selects it -- without touching git or writing anything. Still needs --dest: telling clone from update means checking what's already on disk

run also prints a live <forge>: done/total progress line to stderr while it works, redrawn in place — only when stderr is a terminal, since a carriage-return-redrawn line is unreadable once piped or redirected into a file or CI log.

backup-git-repos list runs the same discovery and filtering without cloning anything, which is the fast way to check a config is picking up what you expect. run --dry-run goes a step further: same idea, but it also says what each repository would do once --dest is factored in.

backup-git-repos config init [--force] writes the starter config described under Configuration.

License

This project is licensed under the GNU General Public License v3.0 — see the LICENSE file for details.

Contributing

Contributions are welcome. Open a pull request.

Read CONTRIBUTING.md first. It covers the setup, how the test suites are split, and what each linter is for.

Documentation

Overview

Package backup mirrors git repositories out of self-hosted forges.

A backup has to survive the forge going away, so what it keeps is a bare mirror clone of every repository: every branch, every tag, and the namespace folder structure the forge used, refreshed in place on later runs. Archived and active repositories can be selected separately, and either set can be written out as a tar.gz alongside the mirror.

Index

Constants

View Source 
const (
	TestActiveRepoPath   = "team/active-repo"
	TestArchivedRepoPath = "team/archived-repo"
	TestEmptyRepoPath    = "team/empty-repo"
)

The paths every Lister's and Runner's fixture data is expected to seed. An adapter's own fixture-seeding driver (the fake, Forgejo, GitLab) is responsible for creating repositories under these exact paths, which is what lets the suites below run unchanged against any of them.

Variables

View Source 
var ErrAmbiguousToken = errors.New("set exactly one of token or token_env")

ErrAmbiguousToken means a forge set both token and token_env. Only one can win silently, and picking one without saying so is how a token meant to override the other quietly gets ignored -- config validation fails instead, the same as issue #9's SSH-key-vs-token rule.

View Source 
var ErrBadArchive = errors.New("archive must be one of: none, all, active, archived")

ErrBadArchive means --archive wasn't one of none, all, active, or archived.

View Source 
var ErrBadState = errors.New("state must be one of: all, active, archived")

ErrBadState means --state wasn't one of all, active, or archived.

View Source 
var ErrGitNotFound = errors.New("git not found on PATH")

ErrGitNotFound means git isn't on PATH.

View Source 
var ErrMissingToken = errors.New("token environment variable not set")

ErrMissingToken means a forge's token_env names an environment variable that isn't set.

Functions

func Archive 

func Archive(dir, out string) error

Archive writes a gzipped tar of the mirror at dir to out, atomically: it writes to out+".tmp" first and renames over out only on success, so a crash mid-write never leaves a truncated archive where a good one was.

Every entry is written under a top-level directory named after dir's own base name, so "tar xzf out" leaves a self-contained "<name>.git/" a caller can git clone directly rather than scattering the mirror's contents into whatever directory they happened to extract into.

Entries are read through an os.Root rooted at dir, which is what stops a symlink inside the mirror from making the walk reference anything outside it. Every entry's ModTime, Uid, Gid, Uname and Gname are zeroed, and entries are written in sorted path order, so archiving an unchanged mirror twice produces a byte-identical file -- which is what makes rsync or deduplication of the backup tree cheap.

func NewRootCommand 

func NewRootCommand(version string, newRunner NewRunner) *cobra.Command

NewRootCommand builds the backup-git-repos command line. Each RunE stays a thin shell: parse and validate the flags, then call into runBackup, which knows nothing about cobra.

func TestBackup 

func TestBackup(t *testing.T, run TestDriver)

TestBackup runs the specification every forge's backup pipeline must satisfy, in domain terms: it talks only about a destination directory and the state it asks for, so the same spec runs unchanged against a fake or a real forge.

func TestLister 

func TestLister(t *testing.T, newLister func(t *testing.T) Lister)

TestLister runs the behaviour every Lister must satisfy, whether it's backed by a fake or a real forge. Exported here, rather than living in a _test.go file, so an adapter under internal/ can run the same suite against itself -- otherwise an unexported helper in this package's own tests would be unreachable from anywhere that isn't this package.

Types

type ArchiveSelection 

type ArchiveSelection int

ArchiveSelection selects which repositories also get written out as a tar.gz alongside their mirror, in addition to being mirrored. 

const (
	ArchiveNone ArchiveSelection = iota
	ArchiveAll
	ArchiveActive
	ArchiveArchived
)

The values ArchiveSelection can hold.

func ParseArchive 

func ParseArchive(s string) (ArchiveSelection, error)

ParseArchive parses a --archive flag value.

type Config 

type Config struct {
	Dest   string        `yaml:"dest"`
	Forges []ForgeConfig `yaml:"forges"`
}

Config is the top-level shape of the YAML config file.

func LoadConfig 

func LoadConfig(path string) (Config, error)

LoadConfig reads and validates the config file at path: every forge's kind must be one this build supports, and every forge's token_env must name a set environment variable. Failing before touching the network beats a 401 partway through a run.

type ForgeConfig 

type ForgeConfig struct {
	Name         string `yaml:"name"`
	Kind         string `yaml:"kind"`
	URL          string `yaml:"url"`
	TokenEnv     string `yaml:"token_env"`
	TokenLiteral string `yaml:"token"`
	Token        string `yaml:"-"`

	// SkipMirrors excludes repositories a forge reports as mirrors of an
	// external upstream from both listing and mirroring. Only the forgejo
	// adapter interprets it today; other kinds ignore it.
	SkipMirrors bool `yaml:"skip_mirrors"`
}

ForgeConfig is one forge entry from the config file. Token is resolved during LoadConfig, either from the environment variable named by TokenEnv or copied straight from TokenLiteral -- never read as-is from the struct the YAML unmarshals into, so every other field reaching a Runner has gone through the same validation regardless of which form the file used.

type Lister 

type Lister interface {
	ListRepos(ctx context.Context, state State) ([]Repo, error)
}

Lister lists the repositories a forge holds, filtered by state.

type Mirror 

type Mirror struct {
	GitPath string
}

Mirror keeps a bare mirror clone of a repository up to date on disk. The zero value works: it resolves git from PATH.

func (Mirror) Sync 

func (m Mirror) Sync(ctx context.Context, r Remote, dir string) error

Sync creates a bare mirror at dir if it doesn't exist yet, or refreshes an existing one otherwise. A refresh prunes refs the remote no longer has.

type Mirrorer 

type Mirrorer interface {
	Sync(ctx context.Context, r Remote, dir string) error
}

Mirrorer keeps a repository's bare mirror up to date on disk.

type NewRunner 

type NewRunner func(ForgeConfig) (Runner, error)

NewRunner builds the Runner for a configured forge. Implementations live with their adapters under internal/, so this package -- which every adapter imports for Repo, Lister and the rest -- can't reference them directly without a cycle. The composition root (main) supplies one.

type Options 

type Options struct {
	Dest        string
	State       State
	Archive     ArchiveSelection
	ArchiveDir  string
	Concurrency int
	Timeout     time.Duration
	Log         *slog.Logger

	// Progress, if set, is called every time a repository is skipped or
	// finishes mirroring (successfully or not), reporting how many of the
	// total have been accounted for so far. Run serializes calls to it, so
	// it doesn't need its own locking even though repositories are
	// mirrored concurrently.
	Progress func(done, total int)
}

Options configures a Run.

type Remote 

type Remote struct {
	CloneURL   string
	AuthHeader string
}

Remote is a repository's clone URL and, where the forge needs one, the value of the Authorization header to send with it. Keeping the credential in a header rather than the URL is what keeps it out of the mirror's own .git/config -- a URL embedded there would leave the token in the clear in every mirror on disk.

type Remoter 

type Remoter interface {
	Remote(repo Repo) Remote
}

Remoter builds the clone Remote for a repository.

type Repo 

type Repo struct {
	Path     string
	Archived bool
	Empty    bool
}

Repo is a repository as reported by a forge: where it lives in the forge's namespace, and whether it's archived or carries no commits yet.

type Result 

type Result struct {
	Synced   int
	Skipped  int
	Failed   int
	Archived int
}

Result reports what a Run did.

type Runner 

type Runner struct {
	Lister   Lister
	Mirrorer Mirrorer
	Remoter  Remoter
}

Runner backs up one forge: it lists repositories, then mirrors each one into a destination tree that mirrors the forge's own namespace structure.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, opts Options) (Result, error)

Run lists repositories filtered by opts.State and mirrors each one, skipping repositories with no refs -- a mirror of one would just confuse the next refresh. It stays synchronous even though it mirrors up to opts.Concurrency repositories at once: the call blocks until every repo has been attempted, and the caller decides whether to run it concurrently with anything else. One repository failing is logged and doesn't stop the rest; Result.Failed is how the caller finds out afterwards.

type State 

type State int

State selects which repositories a Lister returns. 

const (
	StateAll State = iota
	StateActive
	StateArchived
)

The states a Lister can be asked to filter on.

func ParseState 

func ParseState(s string) (State, error)

ParseState parses a --state flag value.

func (State) String 

func (s State) String() string

type TestDriver 

type TestDriver func(ctx context.Context, opts Options) (Result, error)

TestDriver runs a backup the way Runner.Run does, whether that's against an in-memory fake or a real forge container.

type UnknownKindError 

type UnknownKindError struct {
	Kind string
}

UnknownKindError means a forge's kind isn't one backup-git-repos knows how to talk to.

func (*UnknownKindError) Error 

func (e *UnknownKindError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
backup-git-repos command
Command backup-git-repos mirrors git repositories out of self-hosted forges.
 Command backup-git-repos mirrors git repositories out of self-hosted forges.
internal
forgejo
Package forgejo lists and mirrors repositories from a self-hosted Forgejo (or Gitea) instance over its REST API.
 Package forgejo lists and mirrors repositories from a self-hosted Forgejo (or Gitea) instance over its REST API.
github
Package github lists and mirrors repositories from GitHub.com over its REST API.
 Package github lists and mirrors repositories from GitHub.com over its REST API.
gitlab
Package gitlab lists and mirrors repositories from a self-hosted GitLab instance over its REST API.
 Package gitlab lists and mirrors repositories from a self-hosted GitLab instance over its REST API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.