db

package
v0.0.0-...-2faba83 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Index

Constants

View Source
const DefaultPartitionRetentionDays = 365

DefaultPartitionRetentionDays is the age threshold used when cleaning up managed PostgreSQL partitions.

Variables

View Source
var PostgresFunctions = []PostgresFunction{
	{
		Name:       "job_results",
		Definition: jobResultFunction,
	},
	{
		Name:       "test_results",
		Definition: testResultFunction,
	},
}
View Source
var PostgresMatViews = []PostgresView{}

TODO: for historical sippy we need to specify the pinnedDate and not use NOW

View Source
var PostgresViews = []PostgresView{}

PostgresViews are regular, non-materialized views:

Functions

func CopyToTempTable

func CopyToTempTable[T any](
	ctx context.Context,
	conn PgxSession,
	tempTable string,
	rows []T,
	cols []TempColumn[T],
) (cleanup func(), err error)

func DefaultApplicationName

func DefaultApplicationName() (string, error)

DefaultApplicationName returns the PostgreSQL application name used when one is not explicitly configured. Kubernetes sets HOSTNAME to the pod name.

func ExcludeReleases

func ExcludeReleases(releases, excluded []string) []string

ExcludeReleases removes explicitly excluded releases while preserving the supplied order.

func IsSuiteImportable

func IsSuiteImportable(name string) bool

IsSuiteImportable checks if a suite name should be imported based on the explicit testSuites list or dynamic patterns.

func ParseGormLogLevel

func ParseGormLogLevel(logLevel string) (gormlogger.LogLevel, error)

func RefreshByPhase

func RefreshByPhase(matviews []PostgresView, refreshFn func([]PostgresView))

RefreshByPhase groups matviews by RefreshPhase and calls refreshFn for each phase in order. All matviews in a phase are passed to refreshFn together (the caller is responsible for concurrent execution within a phase).

Types

type DB

type DB struct {
	DB *gorm.DB

	// BatchSize is used for how many insertions we should do at once. Postgres supports
	// a maximum of 2^16 records per insert.
	BatchSize int

	// GoparPartitions provides partition creation/management operations
	GoparPartitions *partitioning.DB_PARTITIONS
}

func New

func New(dsn string, logLevel gormlogger.LogLevel, opts ...Option) (*DB, error)

func (*DB) CleanupPartitions

func (d *DB) CleanupPartitions(dryRun bool) (detached, dropped int, err error)

CleanupPartitions performs the full partition lifecycle cleanup: 1. Detaches partitions older than DefaultPartitionRetentionDays 2. Drops detached partitions older than DefaultPartitionRetentionDays

Detached partitions could be held and reattached if needed, but no holding period is configured by default.

Parameters:

  • dryRun: If true, only preview what would be done

Returns the number of partitions detached and dropped.

func (*DB) CopyFrom

func (d *DB) CopyFrom(ctx context.Context, table string, columns []string, rowSrc pgx.CopyFromSource) (int64, error)

CopyFrom uses the PostgreSQL COPY protocol to perform bulk data insertion, streaming all rows in a single protocol-level operation. Callers provide a pgx.CopyFromSource (e.g. pgx.CopyFromRows or pgx.CopyFromSlice) so they can choose whether to pre-allocate rows or generate them lazily.

Columns not listed receive their DEFAULT (e.g. serial id, deleted_at NULL).

func (*DB) DetachOldPartitions

func (d *DB) DetachOldPartitions(retentionDays int, dryRun bool) (int, error)

DetachOldPartitions detaches partitions older than the specified retention period. This is a safer alternative to immediate deletion - detached partitions can be archived or reviewed before permanent deletion.

Parameters:

  • retentionDays: Age threshold in days (e.g., 100 means detach partitions older than 100 days)
  • dryRun: If true, only preview what would be detached

Returns the total number of partitions detached across all tables.

func (*DB) DropDetachedPartitions

func (d *DB) DropDetachedPartitions(retentionDays int, dryRun bool) (int, error)

DropDetachedPartitions drops partition tables whose partition date is older than the retention period. Callers should detach matching attached partitions first. This permanently deletes the data.

Parameters:

  • retentionDays: Minimum partition age in days (e.g., 110 removes partitions dated more than 110 days ago)
  • dryRun: If true, only preview what would be dropped

Returns the total number of partitions dropped across all tables.

func (*DB) EnsurePartitions

func (d *DB) EnsurePartitions(releases []string, startDate, endDate time.Time, dryRun bool) (int, error)

EnsurePartitions creates missing partitions for all managed partitioned tables. It uses LIST→RANGE nested partitioning where:

  • Level 1: LIST partition by release (e.g., "4.17", "4.18")
  • Level 2: RANGE sub-partition by timestamp (daily granularity)

Parameters:

  • releases: List of releases to create partitions for (e.g., ["4.17", "4.18", "4.19"])
  • startDate: Start date for partition creation
  • endDate: End date for partition creation
  • dryRun: If true, only preview what would be created

Returns the total number of partitions created across all tables.

func (*DB) PartitionedTables

func (d *DB) PartitionedTables() []string

PartitionedTables returns the list of tables that are partitioned and managed by gopar partition lifecycle management.

func (*DB) UpdateSchema

func (d *DB) UpdateSchema(reportEnd *time.Time) error

type Option

type Option func(*options)

func WithPartitionwise

func WithPartitionwise(enable bool) Option

type PgxSession

type PgxSession interface {
	Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
	Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
	CopyFrom(ctx context.Context, tableName pgx.Identifier, columnNames []string, rowSrc pgx.CopyFromSource) (int64, error)
}

PgxSession is satisfied by both *pgx.Conn and pgx.Tx.

type PostgresFunction

type PostgresFunction struct {
	Name       string
	Definition string
}

type PostgresView

type PostgresView struct {
	// Name is the name of the materialized view in postgres.
	Name string
	// Definition is the material view definition.
	Definition string
	// ReplaceStrings is a map of strings we want to replace in the create view statement, allowing for re-use.
	ReplaceStrings map[string]string
	// IndexColumns are the columns to create a unique index for. Will be named idx_[Name] and automatically
	// replaced if changes are made to these values. IndexColumns are required as we need them defined to be able to
	// refresh materialized views concurrently. (avoiding locking reads for several minutes while we update)
	IndexColumns []string
	// AdditionalIndexes are non-unique indexes to create on the materialized view for query performance.
	// Each entry is a raw column expression (e.g. "release, timestamp DESC") and will be named
	// idx_[Name]_[sequence].
	AdditionalIndexes []string
	// RefreshPhase controls the order in which matviews are refreshed. All matviews
	// in phase 0 refresh first (concurrently), then all in phase 1, and so on.
	// Use this when a matview reads from another matview and needs it to be up-to-date.
	// The default zero value means phase 0.
	RefreshPhase int
}

type SchemaHashType

type SchemaHashType string

type TempColumn

type TempColumn[T any] struct {
	Name  string
	Type  string
	Value func(*T) any
}

Directories

Path Synopsis
Package infrafailure records infrastructure-failure job runs and removes their contribution from Sippy's pre-aggregated summary tables.
Package infrafailure records infrastructure-failure job runs and removes their contribution from Sippy's pre-aggregated summary tables.
These types are used to decode information from ci-search, but we don't want to expose these for anyone else.
These types are used to decode information from ci-search, but we don't want to expose these for anyone else.
Package verify provides read-only storage adapters and pure comparisons for checking the integrity of Sippy's daily data pipeline.
Package verify provides read-only storage adapters and pure comparisons for checking the integrity of Sippy's daily data pipeline.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL