Documentation
¶
Overview ¶
Package transferstore is the durable persistence layer for the process-wide upload architecture (issue 184). It manages the transfer_files table (one row per source/PAR2 file, pointing at an immutable manifest) and the verification_failures table (only articles that failed a STAT check, with database leases so verification survives crashes).
Successful articles are deliberately never stored: a large upload can contain millions, and per-article rows would overwhelm SQLite.
Index ¶
- Constants
- type Store
- func (s *Store) AddFailure(ctx context.Context, f VerificationFailure) error
- func (s *Store) ClaimDueFailures(ctx context.Context, owner string, leaseDur time.Duration, limit int, ...) ([]VerificationFailure, error)
- func (s *Store) CountFailures(ctx context.Context, transferID, fileID, state string) (int, error)
- func (s *Store) DeleteFilesByTransfer(ctx context.Context, transferID string) error
- func (s *Store) GetCompletedItemNZBPath(ctx context.Context, completedItemID string) (string, error)
- func (s *Store) GetFile(ctx context.Context, transferID, fileID string) (TransferFile, error)
- func (s *Store) ListDueFiles(ctx context.Context, now time.Time, limit int) ([]TransferFile, error)
- func (s *Store) ListFilesByTransfer(ctx context.Context, transferID string) ([]TransferFile, error)
- func (s *Store) MarkUploaded(ctx context.Context, transferID, fileID string, ...) error
- func (s *Store) MigrateLegacyPendingChecks(ctx context.Context) (int, error)
- func (s *Store) ReclaimExpiredLeases(ctx context.Context, now time.Time) (int64, error)
- func (s *Store) SetCleanupPolicy(ctx context.Context, transferID, fileID, policy string) error
- func (s *Store) SetCompletedItemForTransfer(ctx context.Context, transferID, completedItemID string) error
- func (s *Store) SetCompletedItemVerificationStatus(ctx context.Context, completedItemID, status string) error
- func (s *Store) SetUploadState(ctx context.Context, transferID, fileID, state string, postedAt *time.Time) error
- func (s *Store) SetVerificationState(ctx context.Context, transferID, fileID, state string, nextCheckAt *time.Time, ...) error
- func (s *Store) UpdateFailureAfterCheck(ctx context.Context, f VerificationFailure) error
- func (s *Store) UpsertFile(ctx context.Context, f TransferFile) error
- type TransferFile
- type VerificationFailure
Constants ¶
const ( StatePlanned = "planned" StateUploading = "uploading" StateUploaded = "uploaded" StateVerifying = "verifying" StateVerified = "verified" StateVerificationFailed = "verification_failed" )
Upload / verification states for a transfer file.
const ( FailurePending = "pending" FailureReposted = "reposted" FailureResolved = "resolved" FailureFailed = "failed" )
Verification-failure states.
const (
CleanupDeleteOriginal = "delete_original"
)
Cleanup policy markers stored on transfer_files.cleanup_policy. Empty means retain the source. They drive the post-verification cleanup.
const LegacyFileID = "legacy"
LegacyFileID is the synthetic file id used for verification_failures migrated from the pre-durable pending_article_checks table. These records have no manifest, so they are STAT-only (article_index = -1, never re-posted).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store provides durable access to transfer files and verification failures.
func New ¶
New returns a Store backed by db. The transfer_files and verification_failures tables must already exist (migration 007).
func (*Store) AddFailure ¶
func (s *Store) AddFailure(ctx context.Context, f VerificationFailure) error
AddFailure records a failed article for later re-posting/re-checking. Records are unique per (transfer_id, file_id, message_id); duplicates are ignored.
func (*Store) ClaimDueFailures ¶
func (s *Store) ClaimDueFailures(ctx context.Context, owner string, leaseDur time.Duration, limit int, now time.Time) ([]VerificationFailure, error)
ClaimDueFailures atomically leases up to limit pending failures whose next_attempt_at has passed and whose lease is free or expired, marking them owned by owner until now+leaseDur. The claimed rows are returned. Using a transaction makes the select-then-update atomic so concurrent workers (or instances) never claim the same row.
func (*Store) CountFailures ¶
CountFailures returns the number of failure rows for a transfer file in the given state ("" = any state).
func (*Store) DeleteFilesByTransfer ¶
DeleteFilesByTransfer removes all transfer_files rows for a transfer, used after post-verification cleanup completes.
func (*Store) GetCompletedItemNZBPath ¶
func (s *Store) GetCompletedItemNZBPath(ctx context.Context, completedItemID string) (string, error)
GetCompletedItemNZBPath returns the NZB path recorded for a completed item, used by post-verification cleanup to run the post-upload script.
func (*Store) ListDueFiles ¶
ListDueFiles returns files awaiting their first verification check (verification_state = uploaded, next_check_at <= now), oldest first.
func (*Store) ListFilesByTransfer ¶
ListFilesByTransfer returns all files for a transfer.
func (*Store) MarkUploaded ¶
func (s *Store) MarkUploaded(ctx context.Context, transferID, fileID string, postedAt, nextCheckAt time.Time) error
MarkUploaded records that a file finished uploading: upload_state and verification_state both become "uploaded", posted_at is set, and next_check_at is when the first verification check is due (posted_at + post_check.delay).
func (*Store) MigrateLegacyPendingChecks ¶
MigrateLegacyPendingChecks moves any rows from the pre-durable pending_article_checks table into verification_failures as STAT-only records (keyed by transfer_id=completed_item_id, file_id=legacy, article_index=-1), so the durable verification service continues checking them. The legacy rows are deleted in the same transaction. Idempotent (returns 0 once empty), and a no-op when the legacy table does not exist.
func (*Store) ReclaimExpiredLeases ¶
ReclaimExpiredLeases clears leases whose expiry has passed so the work can be picked up again after a crash. Returns the number of rows reclaimed.
func (*Store) SetCleanupPolicy ¶
SetCleanupPolicy records the cleanup policy for a file (e.g. delete_original), read by the post-verification cleanup.
func (*Store) SetCompletedItemForTransfer ¶
func (s *Store) SetCompletedItemForTransfer(ctx context.Context, transferID, completedItemID string) error
SetCompletedItemForTransfer links a transfer's files to the completed_items row created for the upload, so verification status can be reflected back.
func (*Store) SetCompletedItemVerificationStatus ¶
func (s *Store) SetCompletedItemVerificationStatus(ctx context.Context, completedItemID, status string) error
SetCompletedItemVerificationStatus updates the verification_status of a completed item (the user-facing queue row) to reflect durable verification.
func (*Store) SetUploadState ¶
func (s *Store) SetUploadState(ctx context.Context, transferID, fileID, state string, postedAt *time.Time) error
SetUploadState updates a file's upload state and (optionally) posted_at.
func (*Store) SetVerificationState ¶
func (s *Store) SetVerificationState(ctx context.Context, transferID, fileID, state string, nextCheckAt *time.Time, lastErr string) error
SetVerificationState updates a file's verification state, next check time and last error in one statement.
func (*Store) UpdateFailureAfterCheck ¶
func (s *Store) UpdateFailureAfterCheck(ctx context.Context, f VerificationFailure) error
UpdateFailureAfterCheck records the outcome of a check/re-post attempt: the new state, counts, next attempt time and last error, and releases the lease.
func (*Store) UpsertFile ¶
func (s *Store) UpsertFile(ctx context.Context, f TransferFile) error
UpsertFile inserts or replaces a transfer_files row, refreshing updated_at.
type TransferFile ¶
type TransferFile struct {
TransferID string
FileID string
CompletedItemID string
ManifestPath string
ManifestVersion int
SourcePath string
FileRole string
ArticleCount int
UploadState string
VerificationState string
PostedAt *time.Time
NextCheckAt *time.Time
CleanupPolicy string
LastError string
}
TransferFile is one row of the transfer_files table.
type VerificationFailure ¶
type VerificationFailure struct {
ID int64
TransferID string
FileID string
ArticleIndex int
MessageID string
Groups []string
RepostCount int
DeferredCount int
State string
NextAttemptAt time.Time
LeaseOwner string
LeaseExpiresAt *time.Time
LastError string
LastCheckedAt *time.Time
}
VerificationFailure is one row of the verification_failures table.
Source Files