gitpusher

package
v0.1.23-oss Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2026 License: Apache-2.0 Imports: 52 Imported by: 0

README

Background

Git-Pusher is an ABFS component which is responsible for pushing the source code from git repositories to the ABFS remote server.

It is a replacement for upload-daemon which was based on a static configuration. Git-Pusher reads the configuration from the ABFS remote server and updates the internal states based on the config.

It supports multiple manifest-based repositories and multiple independent git projects.

Git-Pusher Architecture

Git-Pusher provides flexibility and scalability by supporting the concept of a pool of pushers. Each pusher is an independent process that can push a subset of projects defined in that pool. The subset is calculated using a consistent hashing algorithm (Rendezvous Hashing) based on the hash of the project name and the number of pushers in the pool.

Git-Pusher Configuration

The Git-Pushers is configured via a YAML format on the remote server for the entire pool. It contains the information about the pool size, the list of pushers inside the pool, list of manifests for repos, and the configuration for each project.

The config can be accessed or edited using a hash-path. By default, the pool is default and the project is config, therefore the config can be edited using:

abfs edit config:git-pusher/default # Use --add/-a for the first time

The format of the config is as follows. The root object is the GitPusherPoolConfig.

Here is a sample config:

$ abfs cat config:git-pusher/default

name: default
pushers-count: 3
defaults:
    port: 9998

pushers:
  - name: gerrit-aosp-0
  - name: gerrit-aosp-1
  - name: gerrit-aosp-2

repos:
  - manifest:
      url: https://android.googlesource.com/platform/manifest
      files:
        - branch: main
          file: default.xml

projects:
  - url: github.com/bazil/fuse
  - url: android.googlesource.com/platform/build/soong
    push-interval: 5s
Root Configuration (GitPusherPoolConfig)
Field Type Description
name string Name of the pusher pool.
defaults GitPusherDefaultsConfig Default configuration for pushers.
pushers-count int Total number of pushers in the pool.
pushers list of GitPusherConfig Configuration for individual pushers.
repos list of RepoConfig Configuration for repositories to watch (manifests).
projects list of ProjectConfig Configuration for individual projects.
GitPusherDefaultsConfig

Defines default configuration for pushers. These can be overriden per pusher.

Field Type Description
port int Listening port of the Git pusher.
max-remote-pushes int 100
max-projects-syncs int 10
max-memory-usage float64 0
GitPusherConfig

Defines a specific pusher instance.

Field Type Default Description
name string - Unique name of the pusher.
index int The position of the item. Index of this pusher (0 to pushers-count - 1).
inactive bool false If true, this pusher is disabled.
url string - URL of the Git pusher.
port int - Listening port of the Git pusher.
max-remote-pushes int 100 Maximum concurrent pushes to the remote.
max-projects-syncs int 10 Maximum concurrent project syncs.
max-memory-usage float64 0 Maximum memory usage percentage (0-100) to self throttle.
RepoConfig

Defines a repository to watch, typically a manifest repository that points to other projects.

Field Type Description
manifest ManifestConfig Configuration for the manifest file.
project-defaults FetchConfig Default fetch settings for projects found in the manifest.
overrides map of string -> OverrideFetchConfig Config overrides for specific projects (non-zero values replace project-defaults)
ManifestConfig
Field Type Default Description
url string - URL of the manifest repository.
server string - Server address (if URL is not full).
project string - Project path (if URL is not full).
files list of ManifestFile - List of manifest files to include.
check-interval duration 60s Interval to check for manifest updates.
ManifestFile
Field Type Description
branch string Branch to look for the manifest file.
file string Path to the manifest XML file.
ProjectConfig

Defines an individual project to sync.

Field Type Description
server string Server address.
project string Project path.
url string Full URL (alternative to server/project).
(inline) FetchConfig Inherits all fields from FetchConfig.
OverrideFetchConfig
Field Type Default Description
exclude bool false If true, this project will be excluded.
regex bool false If true, the key is a regex to match project name.
(inline) FetchConfig Inherits all fields from FetchConfig.
FetchConfig

Shared configuration for fetching and syncing settings.

Field Type Default Description
fetch-ref-spec list of string refs/*:refs/* Refspecs to fetch.
negative-fetch-ref-spec list of string - Refspecs to exclude (must start with ^).
push-interval duration 60s Interval between pushes/syncs.
pack-push-interval duration 30m Interval between pushing the git pack files.
lfs bool false Enable LFS support.
workers int 20 Number of workers for DAG sync.
packing-strategy none, commits, trees, blobs-list, blobs trees The strategy for pushing the git pack files.

Notes

  • Durations: strings like "60s", "1m", "1h".
  • URLs: If url is missing, it is constructed from defaultProtocol (https) + server + project.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrProjectNotFound = errors.New("project not found")

Functions

This section is empty.

Types

type FetchConfig

type FetchConfig struct {
	FetchRefSpecs    []string                `yaml:"fetch-ref-spec"`
	PushInterval     time.Duration           `yaml:"push-interval"`
	LFS              bool                    `yaml:"lfs"`
	Workers          int                     `yaml:"workers"`
	PackPushInterval time.Duration           `yaml:"pack-push-interval"`
	PackingStrategy  binding.PackingStrategy `yaml:"packing-strategy"`
}

func (*FetchConfig) Override

func (f *FetchConfig) Override(g *FetchConfig)

type Git

type Git interface {
	AllObjects(repoPath string) ([]entities.Hash, error)
	RevListAll(repoPath string) ([]entities.Hash, error)
	PackObjects(repoPath string, prefix string, objects []entities.Hash) (string, error)
	AllCommitsDates(repoPath string) ([]entities.Hash, []time.Time, error)
	CommitTreeHashes(root string, commits []entities.Hash) ([]entities.Hash, error)
	CommitsFileObjects(root string, commits []entities.Hash) ([]entities.Hash, error)
	ObjectTypes(repoPath string, objects []entities.Hash) ([]string, error)
}

type GitPusher

type GitPusher struct {
	// contains filtered or unexported fields
}

func NewGitPusher

func NewGitPusher(args *NewGitPusherArgs) *GitPusher

func (*GitPusher) Run

func (g *GitPusher) Run(ctx context.Context) error

func (*GitPusher) SyncDump

func (g *GitPusher) SyncDump(ctx context.Context, projName ProjectName, w io.Writer) error

func (*GitPusher) SyncValidate

func (g *GitPusher) SyncValidate(ctx context.Context, projName ProjectName, w io.Writer, nodeStaleTimeout time.Duration) error

type GitPusherConfig

type GitPusherConfig struct {
	Name             string  `yaml:"name"`
	Index            *int    `yaml:"index"`
	Inactive         bool    `yaml:"inactive"`
	URL              string  `yaml:"url"`
	Port             int     `yaml:"port"`
	MaxRemotePushes  int     `yaml:"max-remote-pushes"`
	MaxProjectsSyncs int     `yaml:"max-projects-syncs"`
	MaxMemoryUsage   float64 `yaml:"max-memory-usage"`
}

type GitPusherPoolConfig

type GitPusherPoolConfig struct {
	Name         string             `yaml:"name"`
	PushersCount int                `yaml:"pushers-count"`
	Pushers      []*GitPusherConfig `yaml:"pushers"`
	Repos        []*RepoConfig      `yaml:"repos"`
	Projects     []*ProjectConfig   `yaml:"projects"`
	Defaults     struct {
		NamePattern      string  `yaml:"name-pattern"`
		Port             int     `yaml:"port"`
		MaxRemotePushes  int     `yaml:"max-remote-pushes"`
		MaxProjectsSyncs int     `yaml:"max-projects-syncs"`
		MaxMemoryUsage   float64 `yaml:"max-memory-usage"`
	} `yaml:"defaults"`
}

func (*GitPusherPoolConfig) Instance

func (c *GitPusherPoolConfig) Instance(name string) (*GitPusherConfig, error)

func (*GitPusherPoolConfig) Validate

func (c *GitPusherPoolConfig) Validate() error

Validate checks if the configuration is valid. It also populates some internal fields and default values.

type ManifestConfig

type ManifestConfig struct {
	URL           string         `yaml:"url"`
	Server        string         `yaml:"server"`
	Project       string         `yaml:"project"`
	Files         []ManifestFile `yaml:"files"`
	CheckInterval time.Duration  `yaml:"check-interval"`
}

type ManifestFile

type ManifestFile struct {
	Branch string `yaml:"branch"`
	File   string `yaml:"file"`
}

type NewGitPusherArgs

type NewGitPusherArgs struct {
	InstanceName string
	ConfigHP     *hashpath.HashPath
	GitBin       string
	StoragePath  string
	Servers      []string
	AuthInfo     instance.AuthInfo
}

type PackPusher

type PackPusher struct {
	// contains filtered or unexported fields
}

func NewPackPusher

func NewPackPusher(client remote.RemoteProvider, server, project, gitRoot string, git Git, strategy binding.PackingStrategy) (*PackPusher, error)

func (*PackPusher) Push

func (p *PackPusher) Push(ctx context.Context) error

type ProjectConfig

type ProjectConfig struct {
	Server      string `yaml:"server"`
	Project     string `yaml:"project"`
	URL         string `yaml:"url"`
	FetchConfig `yaml:",inline"`
}

func (*ProjectConfig) Name

func (c *ProjectConfig) Name() ProjectName

func (*ProjectConfig) Validate

func (c *ProjectConfig) Validate() error

type ProjectLister

type ProjectLister interface {
	ListProjects(ctx context.Context) ([]*ProjectConfig, error)
	Stat() RepoStat
}

type ProjectName

type ProjectName string

func (ProjectName) String

func (p ProjectName) String() string

type PushState

type PushState int
const (
	PushStateCreated PushState = iota
	PushStateInitialized
	PushStateSleep
	PushStateFetch
	PushStateDiff
	PushStatePushBlobs
	PushStatePushObjects
	PushStatePushPacks
)

func (PushState) String

func (s PushState) String() string

type Pusher

type Pusher interface {
	Push(ctx context.Context) error
	Stat() PusherStat
	SyncDump(ctx context.Context, w io.Writer)
	SyncValidate(ctx context.Context, w io.Writer, nodeStaleTimeout time.Duration)
}

type PusherStat

type PusherStat struct {
	Name         ProjectName
	Remote       string
	URL          string
	State        string
	UploadStart  string
	LastUpload   string
	SyncDuration float64
	LastError    string
	DagSync      *dagsync.Stats
}

type RepoConfig

type RepoConfig struct {
	Manifest        ManifestConfig `yaml:"manifest"`
	ProjectDefaults FetchConfig    `yaml:"project-defaults"`
	Overrides       []struct {
		Name         string         `yaml:"name"`
		Regex        bool           `yaml:"regex"`
		RegexPattern *regexp.Regexp `yaml:"-"`
		Exclude      bool           `yaml:"exclude"`
		FetchConfig  `yaml:",inline"`
	} `yaml:"overrides"`
}

func (*RepoConfig) Name

func (c *RepoConfig) Name() RepoName

func (*RepoConfig) Validate

func (c *RepoConfig) Validate() error

type RepoName

type RepoName string

func (RepoName) String

func (r RepoName) String() string

type RepoStat

type RepoStat struct {
	Name          RepoName
	Remote        string
	URL           string
	State         string
	FetchStart    string
	FetchDuration float64
	LastSuccess   string
	LastError     string
}

type RepoWatcher

type RepoWatcher struct {
	// contains filtered or unexported fields
}

func (*RepoWatcher) ListProjects

func (m *RepoWatcher) ListProjects(ctx context.Context) (projects []*ProjectConfig, err error)

ListProjects implements ProjectLister.

func (*RepoWatcher) Stat

func (m *RepoWatcher) Stat() RepoStat

Stat implements ProjectLister.

type RepoWatcherState

type RepoWatcherState int
const (
	RepoWatcherCreated RepoWatcherState = iota
	RepoWatcherInitialized
	RepoWatcherSleep
	RepoWatcherFetch
	RepoWatcherManifestRead
)

func (RepoWatcherState) String

func (s RepoWatcherState) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL