Documentation
¶
Overview ¶
Package files provides file discovery, metadata extraction, and thumbnail generation for the photo gallery. It processes image files from the filesystem and stores metadata in the database.
Index ¶
- Variables
- func CheckIfFileModified(ctx context.Context, cpcRo *dbconnpool.CpConn, f *File) (bool, error)
- func CheckIfFileModifiedWithQueries(ctx context.Context, q QueriesForFiles, f *File) (bool, error)
- func DetectMimeType(f *File, imageFile *os.File) error
- func ExtractExifData(f *File, imageFile *os.File) error
- func GenerateThumbnailAndUpdateDbIfNeeded(ctx context.Context, cpcRw *dbconnpool.CpConn, cpcRo *dbconnpool.CpConn, ...) errordeprecated
- func IsImageFile(path string) bool
- func NeedsFolderTileUpdate(ctx context.Context, cpcRo *dbconnpool.CpConn, folderPath string) (bool, error)
- func NeedsThumbnail(ctx context.Context, cpcRo *dbconnpool.CpConn, fileID int64) (bool, error)
- func NewPoolFuncWithProcessor(processor FileProcessor, q queue.Dequeuer[string], normalizedImagesDir string, ...) workerpool.PoolFunc
- func ProcessFile(ctx context.Context, cpcRo *dbconnpool.CpConn, file *File) error
- func ProcessFileWithQueries(ctx context.Context, q QueriesForFiles, file *File) error
- func UpsertThumbnail(ctx context.Context, cpcRw *dbconnpool.CpConn, fileID int64, thumb []byte) (int64, error)
- func ValidateJpegMarkers(buf []byte) bool
- func WalkImageDir(deps *WalkDeps)
- func WriteFileInTx(ctx context.Context, tx *sql.Tx, f *File) error
- type File
- type FileProcessor
- type Importer
- type ImporterFactory
- type ProcessingStats
- type QueriesForFiles
- type ThumbnailTx
- type UnifiedBatcher
- type WalkDeps
Constants ¶
This section is empty.
Variables ¶
var BufPool = gensyncpool.New( func() *[]byte { b := make([]byte, 8192) return &b }, func(*[]byte) {}, )
BufPool is a pool for temporary byte slices to reduce allocations. Uses 8KB to support both MIME type detection and JPEG marker validation.
var UpsertThumbnailTxOnly = func(qtx ThumbnailTx, ctx context.Context, fileID int64, thumb []byte) (int64, error) { thumbnailID, err := qtx.UpsertThumbnailReturningID(ctx, gallerydb.UpsertThumbnailReturningIDParams{ FileID: fileID, SizeLabel: "m", Height: 0, Width: 0, Format: "jpg", CreatedAt: sql.NullInt64{Int64: time.Now().Unix(), Valid: true}, UpdatedAt: sql.NullInt64{Int64: time.Now().Unix(), Valid: true}, }) if err != nil { return 0, err } if err := qtx.UpsertThumbnailBlob(ctx, gallerydb.UpsertThumbnailBlobParams{ ThumbnailID: thumbnailID, Data: thumb, }); err != nil { return 0, err } return thumbnailID, nil }
UpsertThumbnailTxOnly performs the tx-scoped thumbnail upsert operations using the provided ThumbnailTx. This helper does not manage transactions (Begin/Commit/Rollback); callers are responsible for that. Factoring this out makes it trivial to test the logic with a fake ThumbnailTx.
Functions ¶
func CheckIfFileModified ¶
CheckIfFileModified checks if a file on disk has been modified since it was last recorded in the database by comparing its modification time and size.
func CheckIfFileModifiedWithQueries ¶
CheckIfFileModifiedWithQueries is a testable variant of CheckIfFileModified that uses a QueriesForFiles interface instead of a concrete database connection.
func DetectMimeType ¶
DetectMimeType reads the first 8KB of a file to determine its MIME type. For JPEG files, it also validates that the file contains valid JPEG markers to detect poison pill files that would cause infinite loops in imagemeta.
func ExtractExifData ¶
ExtractExifData uses the `imagemeta` library to extract EXIF and other metadata from an image file. It includes a timeout to prevent tight loops on corrupted files that have JPEG magic bytes but no valid markers.
func GenerateThumbnailAndUpdateDbIfNeeded
deprecated
func GenerateThumbnailAndUpdateDbIfNeeded( ctx context.Context, cpcRw *dbconnpool.CpConn, cpcRo *dbconnpool.CpConn, f *File, importerFactory func(conn *sql.Conn, q *gallerydb.CustomQueries) Importer, ) error
GenerateThumbnailAndUpdateDbIfNeeded orchestrates the thumbnail generation and database update process. It ensures the file and its parent folders are recorded in the database, checks if a thumbnail already exists, and if not, generates and stores a new one. It then updates the parent folder's tile image if necessary.
Deprecated: Prefer the WriteFileInTx path (used by the file write batcher), which batches file and thumbnail writes in a single transaction per batch.
func IsImageFile ¶
IsImageFile checks if a file has a common image file extension and is case-insensitive.
func NeedsFolderTileUpdate ¶
func NeedsFolderTileUpdate(ctx context.Context, cpcRo *dbconnpool.CpConn, folderPath string) (bool, error)
NeedsFolderTileUpdate checks if a tile image has already been assigned for a given folder path.
func NeedsThumbnail ¶
NeedsThumbnail checks if a thumbnail for a given file ID already exists.
func NewPoolFuncWithProcessor ¶
func NewPoolFuncWithProcessor(processor FileProcessor, q queue.Dequeuer[string], normalizedImagesDir string, removePrefix func(normalizedDir, path string) (string, error), stats *ProcessingStats) workerpool.PoolFunc
NewPoolFuncWithProcessor returns a worker pool function that uses FileProcessor service. The returned function matches the signature expected by workerpool.StartWorkerPool.
func ProcessFile ¶
ProcessFile checks if a file has been modified since its last processing, then extracts metadata and generates an in-memory thumbnail if needed.
func ProcessFileWithQueries ¶
func ProcessFileWithQueries(ctx context.Context, q QueriesForFiles, file *File) error
ProcessFileWithQueries is a testable variant of ProcessFile that accepts QueriesForFiles. It doesn't depend on *dbconnpool.CpConn, making it easier to test with mock queries.
func UpsertThumbnail ¶
func UpsertThumbnail(ctx context.Context, cpcRw *dbconnpool.CpConn, fileID int64, thumb []byte) (int64, error)
UpsertThumbnail inserts or updates a thumbnail record and its blob data in a single transaction.
func ValidateJpegMarkers ¶
ValidateJpegMarkers checks if the buffer contains valid JPEG markers after SOI. Returns true if at least one valid JPEG marker (0xFF followed by non-0x00/non-0xFF byte) is found within the buffer. This detects poison pill files that would cause infinite loops in the imagemeta JPEG scanner.
func WalkImageDir ¶
func WalkImageDir(deps *WalkDeps)
WalkImageDir recursively scans the images directory and enqueues image file paths for processing. It runs as a background goroutine; the caller should invoke it via `go files.WalkImageDir(deps)`.
func WriteFileInTx ¶
WriteFileInTx performs all database writes for a single processed file within the provided transaction. It handles: UpsertPathChain, DeleteInvalidFileByPath, UpsertExif, UpsertThumbnail (if needed), and UpdateFolderTileChain.
The caller (FlushFunc) manages BeginTx/Commit/Rollback. This function only executes SQL statements within the provided tx.
After writing, thumbnail buffers are returned to the pool. f.Thumbnail will be nil on return.
Types ¶
type File ¶
type File struct {
Ok bool
Exists bool
ImagesDir string
Path string
File gallerydb.File
Thumbnail *bytes.Buffer
Exif gallerydb.UpsertExifParams
Itpc gallerydb.UpsertIPTCParams
XmpProp gallerydb.UpsertXMPPropertyParams
XmpRaw gallerydb.UpsertXMPRawParams
HasValidJpegMarkers bool // Set by DetectMimeType for JPEG files
}
File represents a file being processed, including its metadata and status.
type FileProcessor ¶
type FileProcessor interface {
// ProcessFile processes a file at the given path, extracting metadata and generating
// an in-memory thumbnail. Returns the processed File struct for further operations.
ProcessFile(ctx context.Context, path string) (*File, error)
// ProcessFileWithConn processes a file using an existing database connection.
// This avoids per-file connection Get/Put overhead when the caller manages the connection lifecycle.
ProcessFileWithConn(ctx context.Context, path string, cpcRo *dbconnpool.CpConn) (*File, error)
// CheckIfModified checks if a file has been modified since it was last processed.
CheckIfModified(ctx context.Context, path string) (bool, error)
// GenerateThumbnail generates a thumbnail for the given file and updates the database.
GenerateThumbnail(ctx context.Context, file *File) error
// RecordInvalidFile records a path in the invalid_files table so it can be skipped on future runs.
RecordInvalidFile(ctx context.Context, path string, mtime, size int64, reason string) error
// SubmitFileForWrite submits a fully-processed *File to the write batcher.
// The batcher handles all DB writes (UpsertPathChain, UpsertExif, UpsertThumbnail,
// etc.) asynchronously in a single transaction. Returns ErrFull if the batcher's
// channel is at capacity. There is no fallback on ErrFull; the file will be
// retried on the next discovery run (self-healing).
SubmitFileForWrite(file *File) error
// PendingWriteCount returns the number of files currently enqueued in the
// write batcher and not yet flushed. Used by completion monitors to avoid
// considering processing done before batcher flushes.
PendingWriteCount() int64
// Close flushes any pending batches and shuts down internal workers.
Close() error
}
FileProcessor provides a high-level interface for file processing operations. It abstracts away the details of processing files, checking modifications, and generating thumbnails.
func NewFileProcessor ¶
func NewFileProcessor( dbRoPool, dbRwPool *dbconnpool.DbSQLConnPool, importerFactory ImporterFactory, imagesDir string, unifiedBatcher UnifiedBatcher, ) FileProcessor
NewFileProcessor returns a FileProcessor implementation that uses the given DB pools, importer factory, and images directory.
type Importer ¶
type Importer interface {
UpsertPathChain(ctx context.Context, path string, mtime, size int64, md5 string, phash, width, height int64, mimeType string) (gallerydb.File, error)
UpdateFolderTileChain(ctx context.Context, folderID, tileFileID int64) error
}
Importer is the minimal interface used by application code that needs to interact with the gallery importer. Defining it here allows tests to provide lightweight fakes without depending on the concrete type in internal/gallerylib.
type ImporterFactory ¶
type ImporterFactory func(conn *sql.Conn, q *gallerydb.CustomQueries) Importer
ImporterFactory creates an Importer from a DB connection and custom queries.
type ProcessingStats ¶
type ProcessingStats struct {
TotalFound atomic.Uint64
AlreadyExisting atomic.Uint64
NewlyInserted atomic.Uint64
SkippedInvalid atomic.Uint64
InFlight atomic.Int64
}
ProcessingStats holds statistics about the file processing job.
func (*ProcessingStats) Reset ¶
func (s *ProcessingStats) Reset()
Reset clears all stats counters to zero. Call this when starting a new discovery run.
type QueriesForFiles ¶
type QueriesForFiles interface {
GetFileByPath(ctx context.Context, path string) (gallerydb.File, error)
GetInvalidFileByPath(ctx context.Context, path string) (gallerydb.InvalidFile, error)
UpsertExif(ctx context.Context, p gallerydb.UpsertExifParams) error
GetThumbnailExistsViewByID(ctx context.Context, id int64) (bool, error)
GetFolderTileExistsViewByPath(ctx context.Context, path string) (bool, error)
WithTx(tx *sql.Tx) ThumbnailTx
}
QueriesForFiles is an interface containing the database methods needed by the file processing logic. This abstraction allows for test fakes.
type ThumbnailTx ¶
type ThumbnailTx interface {
UpsertThumbnailReturningID(ctx context.Context, p gallerydb.UpsertThumbnailReturningIDParams) (int64, error)
UpsertThumbnailBlob(ctx context.Context, p gallerydb.UpsertThumbnailBlobParams) error
}
ThumbnailTx contains the minimal transaction-scoped methods used by the thumbnail upsert logic. This interface is used within database transactions to perform thumbnail-related operations atomically.
type UnifiedBatcher ¶
type UnifiedBatcher interface {
SubmitFile(file *File) error
SubmitInvalidFile(params gallerydb.UpsertInvalidFileParams) error
PendingCount() int64
}
UnifiedBatcher is an interface for submitting mixed write types to the App-level batcher. This avoids circular dependency (files importing server).
Source Files