Documentation
¶
Overview ¶
Package fileproc is the file processor that can be used in conjunction with the transformer. It downloads files to the local filesystem using the provided downloader. Probably it's a good idea to use the downloader.Client for this.
Index ¶
- func AvatarPath(u *slack.User) string
- func ExportTokenUpdateFn(token string) func(_ *slack.Channel, msg *slack.Message) error
- func IsValid(f *slack.File) bool
- func IsValidWithReason(f *slack.File) error
- func New(dl Downloader) processor.Filer
- func NewExport(typ source.StorageType, dl Downloader) processor.Filer
- type AvatarProc
- type DeduplicatingFileProcessor
- type Downloader
- type FileGetter
- type FileProcessor
- type NoopDownloader
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AvatarPath ¶
func ExportTokenUpdateFn ¶
ExportTokenUpdateFn returns a function that appends the token to every file URL in the given message.
func IsValidWithReason ¶
func New ¶
func New(dl Downloader) processor.Filer
New creates a new file processor that uses mattermost file naming pattern.
func NewExport ¶
func NewExport(typ source.StorageType, dl Downloader) processor.Filer
NewExport initialises an export file subprocessor based on the given export type. This subprocessor can be later plugged into the [expproc.Conversations] processor.
Types ¶
type AvatarProc ¶
type AvatarProc struct {
// contains filtered or unexported fields
}
func NewAvatarProc ¶
func NewAvatarProc(dl Downloader) AvatarProc
func (AvatarProc) Close ¶
func (a AvatarProc) Close() error
type DeduplicatingFileProcessor ¶ added in v4.1.1
type DeduplicatingFileProcessor struct {
// contains filtered or unexported fields
}
DeduplicatingFileProcessor wraps a FileProcessor and skips downloading files that already exist in the database with the same ID and size.
func NewDeduplicatingFileProcessor ¶ added in v4.1.1
func NewDeduplicatingFileProcessor(inner processor.Filer, db *sqlx.DB, lg *slog.Logger) *DeduplicatingFileProcessor
NewDeduplicatingFileProcessor creates a new file processor that skips downloading files that already exist in the database.
func (*DeduplicatingFileProcessor) Close ¶ added in v4.1.1
func (d *DeduplicatingFileProcessor) Close() error
Close stops the underlying downloader.
func (*DeduplicatingFileProcessor) Files ¶ added in v4.1.1
func (d *DeduplicatingFileProcessor) Files(ctx context.Context, channel *slack.Channel, parent slack.Message, ff []slack.File) error
Files processes files, skipping those that already exist in the database with the same ID and size.
Files are processed one-by-one rather than batched, because:
- Files arrive per-message, so we can't batch across all files (streaming)
- Each file download is independent and can start immediately
- The underlying Filer handles its own batching/parallelism if needed
Note: This makes one DB query per file. For messages with multiple files, this could be optimized to batch the lookup (WHERE ID IN (...) AND SIZE IN (...)), but most messages have 0-2 files so the benefit is limited.
type Downloader ¶
type Downloader interface {
// Download should download the file at the given URL and save it to the
// given path.
Download(fullpath string, url string) error
Stop()
}
Downloader is the interface that wraps the Download method.
func NewDownloader ¶
func NewDownloader(ctx context.Context, enabled bool, cl FileGetter, fsa fsadapter.FS, lg *slog.Logger) (sdl Downloader)
NewDownloader initializes the downloader and returns it, along with a function that should be called to stop it.
type FileGetter ¶
type FileProcessor ¶
type FileProcessor struct {
// contains filtered or unexported fields
}
FileProcessor is the file processor, that downloads files to the path returned by the filepath function. Zero value of this type is not usable.
func NewDump ¶
func NewDump(dl Downloader) FileProcessor
NewDump returns a new Dump File FileProcessor.
func NewWithPathFn ¶
func NewWithPathFn(dl Downloader, fp func(ci *slack.Channel, f *slack.File) string) FileProcessor
NewWithPathFn initialises the file processor.
func (FileProcessor) Close ¶
func (b FileProcessor) Close() error
func (FileProcessor) PathUpdateFunc ¶
func (b FileProcessor) PathUpdateFunc(channelID, threadTS string, mm []slack.Message) error
PathUpdateFunc updates the path in URLDownload and URLPrivateDownload of every file in the given message slice to point to the physical downloaded file location. It can be plugged in the pipeline of Dump.
type NoopDownloader ¶
type NoopDownloader struct{}
func (NoopDownloader) Stop ¶
func (NoopDownloader) Stop()
Source Files