Documentation
¶
Index ¶
- Constants
- Variables
- func CopyToTempTable[T any](ctx context.Context, conn PgxSession, tempTable string, rows []T, ...) (cleanup func(), err error)
- func DefaultApplicationName() (string, error)
- func ExcludeReleases(releases, excluded []string) []string
- func IsSuiteImportable(name string) bool
- func ParseGormLogLevel(logLevel string) (gormlogger.LogLevel, error)
- func RefreshByPhase(matviews []PostgresView, refreshFn func([]PostgresView))
- type DB
- func (d *DB) CleanupPartitions(dryRun bool) (detached, dropped int, err error)
- func (d *DB) CopyFrom(ctx context.Context, table string, columns []string, rowSrc pgx.CopyFromSource) (int64, error)
- func (d *DB) DetachOldPartitions(retentionDays int, dryRun bool) (int, error)
- func (d *DB) DropDetachedPartitions(retentionDays int, dryRun bool) (int, error)
- func (d *DB) EnsurePartitions(releases []string, startDate, endDate time.Time, dryRun bool) (int, error)
- func (d *DB) PartitionedTables() []string
- func (d *DB) UpdateSchema(reportEnd *time.Time) error
- type Option
- type PgxSession
- type PostgresFunction
- type PostgresView
- type SchemaHashType
- type TempColumn
Constants ¶
const DefaultPartitionRetentionDays = 365
DefaultPartitionRetentionDays is the age threshold used when cleaning up managed PostgreSQL partitions.
Variables ¶
var PostgresFunctions = []PostgresFunction{
{
Name: "job_results",
Definition: jobResultFunction,
},
{
Name: "test_results",
Definition: testResultFunction,
},
}
var PostgresMatViews = []PostgresView{}
TODO: for historical sippy we need to specify the pinnedDate and not use NOW
var PostgresViews = []PostgresView{}
PostgresViews are regular, non-materialized views:
Functions ¶
func CopyToTempTable ¶
func CopyToTempTable[T any]( ctx context.Context, conn PgxSession, tempTable string, rows []T, cols []TempColumn[T], ) (cleanup func(), err error)
func DefaultApplicationName ¶
DefaultApplicationName returns the PostgreSQL application name used when one is not explicitly configured. Kubernetes sets HOSTNAME to the pod name.
func ExcludeReleases ¶
ExcludeReleases removes explicitly excluded releases while preserving the supplied order.
func IsSuiteImportable ¶
IsSuiteImportable checks if a suite name should be imported based on the explicit testSuites list or dynamic patterns.
func ParseGormLogLevel ¶
func ParseGormLogLevel(logLevel string) (gormlogger.LogLevel, error)
func RefreshByPhase ¶
func RefreshByPhase(matviews []PostgresView, refreshFn func([]PostgresView))
RefreshByPhase groups matviews by RefreshPhase and calls refreshFn for each phase in order. All matviews in a phase are passed to refreshFn together (the caller is responsible for concurrent execution within a phase).
Types ¶
type DB ¶
type DB struct {
DB *gorm.DB
// BatchSize is used for how many insertions we should do at once. Postgres supports
// a maximum of 2^16 records per insert.
BatchSize int
// GoparPartitions provides partition creation/management operations
GoparPartitions *partitioning.DB_PARTITIONS
}
func (*DB) CleanupPartitions ¶
CleanupPartitions performs the full partition lifecycle cleanup: 1. Detaches partitions older than DefaultPartitionRetentionDays 2. Drops detached partitions older than DefaultPartitionRetentionDays
Detached partitions could be held and reattached if needed, but no holding period is configured by default.
Parameters:
- dryRun: If true, only preview what would be done
Returns the number of partitions detached and dropped.
func (*DB) CopyFrom ¶
func (d *DB) CopyFrom(ctx context.Context, table string, columns []string, rowSrc pgx.CopyFromSource) (int64, error)
CopyFrom uses the PostgreSQL COPY protocol to perform bulk data insertion, streaming all rows in a single protocol-level operation. Callers provide a pgx.CopyFromSource (e.g. pgx.CopyFromRows or pgx.CopyFromSlice) so they can choose whether to pre-allocate rows or generate them lazily.
Columns not listed receive their DEFAULT (e.g. serial id, deleted_at NULL).
func (*DB) DetachOldPartitions ¶
DetachOldPartitions detaches partitions older than the specified retention period. This is a safer alternative to immediate deletion - detached partitions can be archived or reviewed before permanent deletion.
Parameters:
- retentionDays: Age threshold in days (e.g., 100 means detach partitions older than 100 days)
- dryRun: If true, only preview what would be detached
Returns the total number of partitions detached across all tables.
func (*DB) DropDetachedPartitions ¶
DropDetachedPartitions drops partition tables whose partition date is older than the retention period. Callers should detach matching attached partitions first. This permanently deletes the data.
Parameters:
- retentionDays: Minimum partition age in days (e.g., 110 removes partitions dated more than 110 days ago)
- dryRun: If true, only preview what would be dropped
Returns the total number of partitions dropped across all tables.
func (*DB) EnsurePartitions ¶
func (d *DB) EnsurePartitions(releases []string, startDate, endDate time.Time, dryRun bool) (int, error)
EnsurePartitions creates missing partitions for all managed partitioned tables. It uses LIST→RANGE nested partitioning where:
- Level 1: LIST partition by release (e.g., "4.17", "4.18")
- Level 2: RANGE sub-partition by timestamp (daily granularity)
Parameters:
- releases: List of releases to create partitions for (e.g., ["4.17", "4.18", "4.19"])
- startDate: Start date for partition creation
- endDate: End date for partition creation
- dryRun: If true, only preview what would be created
Returns the total number of partitions created across all tables.
func (*DB) PartitionedTables ¶
PartitionedTables returns the list of tables that are partitioned and managed by gopar partition lifecycle management.
type PgxSession ¶
type PgxSession interface {
Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
CopyFrom(ctx context.Context, tableName pgx.Identifier, columnNames []string, rowSrc pgx.CopyFromSource) (int64, error)
}
PgxSession is satisfied by both *pgx.Conn and pgx.Tx.
type PostgresFunction ¶
type PostgresView ¶
type PostgresView struct {
// Name is the name of the materialized view in postgres.
Name string
// Definition is the material view definition.
Definition string
// ReplaceStrings is a map of strings we want to replace in the create view statement, allowing for re-use.
ReplaceStrings map[string]string
// IndexColumns are the columns to create a unique index for. Will be named idx_[Name] and automatically
// replaced if changes are made to these values. IndexColumns are required as we need them defined to be able to
// refresh materialized views concurrently. (avoiding locking reads for several minutes while we update)
IndexColumns []string
// AdditionalIndexes are non-unique indexes to create on the materialized view for query performance.
// Each entry is a raw column expression (e.g. "release, timestamp DESC") and will be named
// idx_[Name]_[sequence].
AdditionalIndexes []string
// RefreshPhase controls the order in which matviews are refreshed. All matviews
// in phase 0 refresh first (concurrently), then all in phase 1, and so on.
// Use this when a matview reads from another matview and needs it to be up-to-date.
// The default zero value means phase 0.
RefreshPhase int
}
type SchemaHashType ¶
type SchemaHashType string
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package infrafailure records infrastructure-failure job runs and removes their contribution from Sippy's pre-aggregated summary tables.
|
Package infrafailure records infrastructure-failure job runs and removes their contribution from Sippy's pre-aggregated summary tables. |
|
These types are used to decode information from ci-search, but we don't want to expose these for anyone else.
|
These types are used to decode information from ci-search, but we don't want to expose these for anyone else. |
|
Package verify provides read-only storage adapters and pure comparisons for checking the integrity of Sippy's daily data pipeline.
|
Package verify provides read-only storage adapters and pure comparisons for checking the integrity of Sippy's daily data pipeline. |