files

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 37 Imported by: 0

Documentation

Overview

Package files provides file discovery, metadata extraction, and thumbnail generation for the photo gallery. It processes image files from the filesystem and stores metadata in the database.

Index

Constants

This section is empty.

Variables

View Source 
var BufPool = gensyncpool.New(
	func() *[]byte {
		b := make([]byte, 8192)
		return &b
	},
	func(*[]byte) {},
)

BufPool is a pool for temporary byte slices to reduce allocations. Uses 8KB to support both MIME type detection and JPEG marker validation.

View Source 
var ErrFolderIndexRebuild = errors.New("file_folder_index rebuild failed")

ErrFolderIndexRebuild is returned (wrapped) by RebuildFileFolderIndex for any failure other than context cancellation. The startup goroutine uses it to decide between Shutdown and keeping the server serving.

View Source 
var UpsertThumbnailTxOnly = func(qtx ThumbnailTx, ctx context.Context, fileID int64, thumb []byte) (int64, error) {
	thumbnailID, err := qtx.UpsertThumbnailReturningID(ctx, gallerydb.UpsertThumbnailReturningIDParams{
		FileID:    fileID,
		SizeLabel: "m",
		Height:    0,
		Width:     0,
		Format:    "jpg",
		CreatedAt: sql.NullInt64{Int64: time.Now().Unix(), Valid: true},
		UpdatedAt: sql.NullInt64{Int64: time.Now().Unix(), Valid: true},
	})
	if err != nil {
		return 0, err
	}

	if err := qtx.UpsertThumbnailBlob(ctx, gallerydb.UpsertThumbnailBlobParams{
		ThumbnailID: thumbnailID,
		Data:        thumb,
	}); err != nil {
		return 0, err
	}
	return thumbnailID, nil
}

UpsertThumbnailTxOnly performs the tx-scoped thumbnail upsert operations using the provided ThumbnailTx. This helper does not manage transactions (Begin/Commit/Rollback); callers are responsible for that. Factoring this out makes it trivial to test the logic with a fake ThumbnailTx.

Functions

func CheckIfFileModified 

func CheckIfFileModified(ctx context.Context, cpcRo *dbconnpool.CpConn, f *File) (bool, error)

CheckIfFileModified checks if a file on disk has been modified since it was last recorded in the database by comparing its modification time and size.

func CheckIfFileModifiedWithQueries 

func CheckIfFileModifiedWithQueries(ctx context.Context, q QueriesForFiles, f *File) (bool, error)

CheckIfFileModifiedWithQueries is a testable variant of CheckIfFileModified that uses a QueriesForFiles interface instead of a concrete database connection.

func DetectMimeType 

func DetectMimeType(f *File, imageFile *os.File) error

DetectMimeType reads the first 8KB of a file to determine its MIME type. For JPEG files, it also validates that the file contains valid JPEG markers to detect poison pill files that would cause infinite loops in imagemeta.

func ExtractExifData 

func ExtractExifData(f *File, imageFile *os.File) error

ExtractExifData uses the `imagemeta` library to extract EXIF and other metadata from an image file. It includes a timeout to prevent tight loops on corrupted files that have JPEG magic bytes but no valid markers.

func GenerateThumbnailAndUpdateDbIfNeeded deprecated 

func GenerateThumbnailAndUpdateDbIfNeeded(
	ctx context.Context,
	cpcRw *dbconnpool.CpConn,
	cpcRo *dbconnpool.CpConn,
	f *File,
	importerFactory func(conn *sql.Conn, q *gallerydb.CustomQueries) Importer,
) error

GenerateThumbnailAndUpdateDbIfNeeded orchestrates the thumbnail generation and database update process. It ensures the file and its parent folders are recorded in the database, checks if a thumbnail already exists, and if not, generates and stores a new one. It then updates the parent folder's tile image if necessary.

Deprecated: Prefer the WriteFileInTx path (used by the file write batcher), which batches file and thumbnail writes in a single transaction per batch.

func IsImageFile 

func IsImageFile(path string) bool

IsImageFile checks if a file has a common image file extension and is case-insensitive.

func NeedsFolderTileUpdate 

func NeedsFolderTileUpdate(ctx context.Context, cpcRo *dbconnpool.CpConn, folderPath string) (bool, error)

NeedsFolderTileUpdate checks if a tile image has already been assigned for a given folder path.

func NeedsThumbnail 

func NeedsThumbnail(ctx context.Context, cpcRo *dbconnpool.CpConn, fileID int64) (bool, error)

NeedsThumbnail checks if a thumbnail for a given file ID already exists.

func NewPoolFuncWithProcessor 

func NewPoolFuncWithProcessor(processor FileProcessor, q queue.Dequeuer[string], normalizedImagesDir string, removePrefix func(normalizedDir, path string) (string, error), stats *ProcessingStats, onFileInserted func(int64)) workerpool.PoolFunc

NewPoolFuncWithProcessor returns a worker pool function that uses FileProcessor service. The returned function matches the signature expected by workerpool.StartWorkerPool.

func ProcessFile 

func ProcessFile(ctx context.Context, cpcRo *dbconnpool.CpConn, file *File) error

ProcessFile checks if a file has been modified since its last processing, then extracts metadata and generates an in-memory thumbnail if needed.

func ProcessFileWithQueries 

func ProcessFileWithQueries(ctx context.Context, q QueriesForFiles, file *File) error

ProcessFileWithQueries is a testable variant of ProcessFile that accepts QueriesForFiles. It doesn't depend on *dbconnpool.CpConn, making it easier to test with mock queries.

func RebuildFileFolderIndex added in v0.15.0 

func RebuildFileFolderIndex(ctx context.Context, rw, ro *dbconnpool.DbSQLConnPool, batcher UnifiedBatcher) error

RebuildFileFolderIndex materializes file_folder_index from the files table without a gallery-wide window query. It streams (file_id, folder_id) pairs on the RO pool into a pre-sized [][2]int64, closes the cursor, releases the RO conn, then computes per-folder navigation columns in Go and submits each row via SubmitFolderIndex to the unified batcher (which INSERTs into file_folder_index_new). After the batcher flushes (inflight reaches 0), it CreateIndexes and Swaps on a fresh RW conn. Swap DROP TABLEs file_folder_index_to_be_dropped before it returns.

The RW conn is Put immediately after CloneEmpty, so Submit and the inflight wait never hold a write lock (the batcher and Swap need it free).

func SetGPSFromStrings added in v0.9.0 

func SetGPSFromStrings(f *File, latStr, lonStr string) error

SetGPSFromStrings parses DMS/decimal GPS strings via coords and assigns f.Exif. Production entry point when coordinates arrive as strings (XMP sidecar properties, import pipelines, etc.). Returns error if either coordinate fails to parse.

func UpsertThumbnail 

func UpsertThumbnail(ctx context.Context, cpcRw *dbconnpool.CpConn, fileID int64, thumb []byte) (int64, error)

UpsertThumbnail inserts or updates a thumbnail record and its blob data in a single transaction.

func ValidateJpegMarkers 

func ValidateJpegMarkers(buf []byte) bool

ValidateJpegMarkers checks if the buffer contains valid JPEG markers after SOI. Returns true if at least one valid JPEG marker (0xFF followed by non-0x00/non-0xFF byte) is found within the buffer. This detects poison pill files that would cause infinite loops in the imagemeta JPEG scanner.

func WalkImageDir 

func WalkImageDir(deps *WalkDeps)

WalkImageDir recursively scans the images directory and enqueues image file paths for processing. It runs as a background goroutine; the caller should invoke it via `go files.WalkImageDir(deps)`.

func WriteFileInTx 

func WriteFileInTx(ctx context.Context, imp *gallerylib.Importer, f *File) error

WriteFileInTx performs all database writes for a single processed file within the provided transaction. It handles: UpsertPathChain, DeleteInvalidFileByPath, UpsertExif, UpsertXMP (raw + properties, if captured), UpsertThumbnail (if needed), and UpdateFolderTileChain.

The caller (FlushFunc) manages BeginTx/Commit/Rollback. This function only executes SQL statements within the provided tx.

imp must be backed by the same *gallerydb.CustomQueries that is bound to tx (i.e. imp.Q is the WithTx(tx) view). Construct ONE imp per batch and reuse it across files so imp's intra-batch folder cache and tiled-dir set persist.

After writing, thumbnail buffers are returned to the pool. f.Thumbnail will be nil on return.

Types

type File 

type File struct {
	Ok                  bool
	Exists              bool
	ImagesDir           string
	Path                string
	File                gallerydb.File
	Thumbnail           *bytes.Buffer
	Exif                gallerydb.UpsertExifParams
	Itpc                gallerydb.UpsertIPTCParams
	XmpProps            []gallerydb.UpsertXMPPropertyParams
	XmpRaw              gallerydb.UpsertXMPRawParams
	HasValidJpegMarkers bool // Set by DetectMimeType for JPEG files
}

File represents a file being processed, including its metadata and status.

func (*File) GobDecode added in v0.3.0 

func (f *File) GobDecode(data []byte) error

GobDecode deserializes File from the gob-safe wire format, reconstructing the Thumbnail bytes.Buffer from the raw bytes. Note: the reconstructed buffer is created via bytes.NewBuffer, not from the thumbnail pool. When cleanupBatchedWriteResources later returns it to the pool via thumbnail.PutBytesBuffer, the pool accepts any *bytes.Buffer.

func (File) GobEncode added in v0.3.0 

func (f File) GobEncode() ([]byte, error)

GobEncode serializes File into a gob-safe wire format. The *bytes.Buffer Thumbnail field is extracted as raw bytes; the caller's object is not mutated.

type FileProcessor 

type FileProcessor interface {
	// ProcessFile processes a file at the given path, extracting metadata and generating
	// an in-memory thumbnail. Returns the processed File struct for further operations.
	ProcessFile(ctx context.Context, path string) (*File, error)

	// ProcessFileWithConn processes a file using an existing database connection.
	// This avoids per-file connection Get/Put overhead when the caller manages the connection lifecycle.
	ProcessFileWithConn(ctx context.Context, path string, cpcRo *dbconnpool.CpConn) (*File, error)

	// CheckIfModified checks if a file has been modified since it was last processed.
	CheckIfModified(ctx context.Context, path string) (bool, error)

	// GenerateThumbnail generates a thumbnail for the given file and updates the database.
	GenerateThumbnail(ctx context.Context, file *File) error

	// RecordInvalidFile records a path in the invalid_files table so it can be skipped on future runs.
	RecordInvalidFile(ctx context.Context, path string, mtime, size int64, reason string) error

	// SubmitFileForWrite submits a fully-processed *File to the write batcher.
	// The batcher handles all DB writes (UpsertPathChain, UpsertExif, UpsertThumbnail,
	// etc.) asynchronously in a single transaction. Returns ErrFull if the batcher's
	// channel is at capacity. There is no fallback on ErrFull; the file will be
	// retried on the next discovery run (self-healing).
	SubmitFileForWrite(file *File) error

	// PendingWriteCount returns the number of files currently enqueued in the
	// write batcher and not yet flushed. Used by completion monitors to avoid
	// considering processing done before batcher flushes.
	PendingWriteCount() int64

	// Close flushes any pending batches and shuts down internal workers.
	Close() error
}

FileProcessor provides a high-level interface for file processing operations. It abstracts away the details of processing files, checking modifications, and generating thumbnails.

func NewFileProcessor 

func NewFileProcessor(
	dbRoPool, dbRwPool *dbconnpool.DbSQLConnPool,
	importerFactory ImporterFactory,
	imagesDir string,
	unifiedBatcher UnifiedBatcher,
) FileProcessor

NewFileProcessor returns a FileProcessor implementation that uses the given DB pools, importer factory, and images directory.

type FolderIndexRow added in v0.15.0 

type FolderIndexRow struct {
	FileID     int64
	FolderID   int64
	ImageIndex int64
	ImageCount int64
	PrevID     sql.NullInt64
	NextID     sql.NullInt64
	FirstID    int64
	LastID     int64
	Generation int64
}

FolderIndexRow is one materialized navigation row for file_folder_index. It carries a Generation that ties it to a specific in-process rebuild so a dque leftover from a previous process (with a different generation) cannot PK-wedge the rebuild's INSERTs on startup.

type Importer 

type Importer interface {
	UpsertPathChain(ctx context.Context, path string, mtime, size int64, md5 string, phash, width, height int64, mimeType string) (gallerydb.File, error)
	UpdateFolderTileChain(ctx context.Context, folderID, tileFileID int64) error
}

Importer is the minimal interface used by application code that needs to interact with the gallery importer. Defining it here allows tests to provide lightweight fakes without depending on the concrete type in internal/gallerylib.

type ImporterFactory 

type ImporterFactory func(conn *sql.Conn, q *gallerydb.CustomQueries) Importer

ImporterFactory creates an Importer from a DB connection and custom queries.

type ProcessingStats 

type ProcessingStats struct {
	TotalFound      atomic.Uint64
	AlreadyExisting atomic.Uint64
	NewlyInserted   atomic.Uint64
	SkippedInvalid  atomic.Uint64
	InFlight        atomic.Int64
}

ProcessingStats holds statistics about the file processing job.

func (*ProcessingStats) GetStats added in v0.9.0 

func (s *ProcessingStats) GetStats() metrics.FileProcessingMetrics

GetStats returns the current file-processing counters in the shape used by the metrics dashboard.

func (*ProcessingStats) Reset 

func (s *ProcessingStats) Reset()

Reset clears all stats counters to zero. Call this when starting a new discovery run.

type QueriesForFiles 

type QueriesForFiles interface {
	GetFileByPath(ctx context.Context, path string) (gallerydb.File, error)
	GetInvalidFileByPath(ctx context.Context, path string) (gallerydb.InvalidFile, error)
	UpsertExif(ctx context.Context, p gallerydb.UpsertExifParams) error
	GetThumbnailExistsViewByID(ctx context.Context, id int64) (bool, error)
	GetFolderTileExistsViewByPath(ctx context.Context, path string) (bool, error)
	WithTx(tx *sql.Tx) ThumbnailTx
}

QueriesForFiles is an interface containing the database methods needed by the file processing logic. This abstraction allows for test fakes.

type ThumbnailTx 

type ThumbnailTx interface {
	UpsertThumbnailReturningID(ctx context.Context, p gallerydb.UpsertThumbnailReturningIDParams) (int64, error)
	UpsertThumbnailBlob(ctx context.Context, p gallerydb.UpsertThumbnailBlobParams) error
}

ThumbnailTx contains the minimal transaction-scoped methods used by the thumbnail upsert logic. This interface is used within database transactions to perform thumbnail-related operations atomically.

type UnifiedBatcher 

type UnifiedBatcher interface {
	SubmitFile(file *File) error
	SubmitFolderIndex(row FolderIndexRow) error
	PendingCount() int64
	FolderIndexInflight() int64
	SetFolderIndexRebuildActive(active bool)
	BumpFolderIndexGeneration() int64
	// SetFolderIndexRebuildScanHeld marks the RO scan cursor open/closed during a
	// rebuild. While held, the infrastructure service skips WAL TRUNCATE
	// checkpoints (the cursor pins the WAL write lock).
	SetFolderIndexRebuildScanHeld(held bool)
}

UnifiedBatcher is an interface for submitting mixed write types to the App-level batcher. This avoids circular dependency (files importing server).

type WalkDeps 

type WalkDeps struct {
	Wg             *sync.WaitGroup
	QSendersActive *atomic.Int64
	Ctx            context.Context
	ImagesDir      string
	Q              queue.Enqueuer[string]
}

WalkDeps holds dependencies for WalkImageDir. Passed by the caller (e.g. App).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.