Documentation
¶
Overview ¶
Package backup is providing commands to inspect a file system (hard-drive, USB, Android, S3) and backup medias to a remote DPhoto storage.
Index ¶
- Constants
- Variables
- type AlbumLookupPort
- type AlbumReport
- type AnalysedMedia
- type AnalysedMediaObserver
- type AnalysedMediaObserverFunc
- type AnalysedMediaObservers
- type Analyser
- type AnalyserDecorator
- type AnalyserDecoratorObserver
- type AnalyserFromMediaDetails
- type ArchiveMediaPort
- type BackingUpMediaRequest
- type BatchBackup
- type BatchScanner
- type CatalogMediaRequest
- type CatalogReference
- type CatalogReferencerObserver
- type CatalogReferencerObserverFunc
- type CatalogReferencerObservers
- type Cataloguer
- type CataloguerFactory
- type CataloguerFilterObserver
- type CataloguerFilterObservers
- type ClosableFoundMedia
- type ConcurrencyParameters
- type DetailsReader
- type DetailsReaderOptions
- type ExtraCounts
- type FoundMedia
- type FullMediaSignature
- type ImageOrientation
- type InMemoryMedia
- type IndexMediaPort
- type InsertMediaPort
- type MediaCounter
- type MediaDetails
- type MediaPath
- type MediaType
- type NopeAnalyserDecorator
- type Options
- func OptionsAnalyserDecorator(analyserDecorator AnalyserDecorator) Options
- func OptionsBatchSize(batchSize int) Options
- func OptionsChannelSize(i int) Options
- func OptionsConcurrentAnalyserRoutines(concurrent int) Options
- func OptionsConcurrentCataloguerRoutines(concurrent int) Options
- func OptionsConcurrentUploaderRoutines(concurrent int) Options
- func OptionsOnlyAlbums(albums ...string) Options
- func OptionsSkipRejects(skip bool) Options
- func OptionsWithListener(listener interface{}) Options
- func OptionsWithRejectDir(rejectDir string) Options
- func ReduceOptions(requestedOptions ...Options) Options
- type PostAnalyseFilter
- type RejectedMediaObserver
- type RejectedMediaObservers
- type Report
- type ScannedFolder
- type SourceVolume
- type TimelinePort
- type TrackAnalysed
- type TrackEvents
- type TrackScanComplete
- type TrackUploaded
Constants ¶
const ( MediaTypeImage MediaType = "IMAGE" MediaTypeVideo MediaType = "VIDEO" MediaTypeOther MediaType = "OTHER" OrientationUpperLeft ImageOrientation = "UPPER_LEFT" OrientationLowerRight ImageOrientation = "LOWER_RIGHT" OrientationUpperRight ImageOrientation = "UPPER_RIGHT" OrientationLowerLeft ImageOrientation = "LOWER_LEFT" )
Variables ¶
var ( ErrAnalyserNoDateTime = errors.New("media must have a date time included in the metadata") ErrAnalyserNotSupported = errors.New("media format is not supported") )
var ( ErrCatalogerFilterMustBeInAlbum = errors.New("media must be in album") ErrCatalogerFilterMustNotAlreadyExists = errors.New("media must not already exists") ErrMediaMustNotBeDuplicated = errors.New("media is present twice in the volume") )
var (
MediaCounterZero = MediaCounter{}
)
var SupportedExtensions = map[string]MediaType{ "jpg": MediaTypeImage, "jpeg": MediaTypeImage, "png": MediaTypeImage, "gif": MediaTypeImage, "webp": MediaTypeImage, "raw": MediaTypeImage, "svg": MediaTypeImage, "eps": MediaTypeImage, "mkv": MediaTypeVideo, "mts": MediaTypeVideo, "avi": MediaTypeVideo, "mp4": MediaTypeVideo, "mpeg": MediaTypeVideo, "mov": MediaTypeVideo, "wmv": MediaTypeVideo, "webm": MediaTypeVideo, }
SupportedExtensions is used by SourceVolume adapters to filter files they find
Functions ¶
This section is empty.
Types ¶
type AlbumLookupPort ¶ added in v1.5.54
type AlbumReport ¶ added in v1.6.35
type AlbumReport struct {
// contains filtered or unexported fields
}
func NewAlbumReport ¶ added in v1.7.0
func NewAlbumReport(mediaType MediaType, count int, size int, isNew bool) *AlbumReport
NewAlbumReport is a convenience method for testing or mocking 'backup' domain
func (*AlbumReport) IsNew ¶ added in v1.7.0
func (c *AlbumReport) IsNew() bool
func (*AlbumReport) OfType ¶ added in v1.6.35
func (c *AlbumReport) OfType(mediaType MediaType) MediaCounter
func (*AlbumReport) Total ¶ added in v1.6.35
func (c *AlbumReport) Total() MediaCounter
type AnalysedMedia ¶
type AnalysedMedia struct {
FoundMedia FoundMedia // FoundMedia is the reference of the file, implementation depends on the VolumeType
Type MediaType // Type is 'photo' or 'video'
Sha256Hash string // Sha256Hash sha256 of the file
Details *MediaDetails // Details are data found within the file (location, date, ...)
}
AnalysedMedia is a FoundMedia to which has been attached its type (photo / video) and other details usually found within the file.
type AnalysedMediaObserver ¶ added in v1.6.26
type AnalysedMediaObserver interface {
OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error
}
type AnalysedMediaObserverFunc ¶ added in v1.6.35
type AnalysedMediaObserverFunc func(ctx context.Context, media *AnalysedMedia) error
func (AnalysedMediaObserverFunc) OnAnalysedMedia ¶ added in v1.6.35
func (a AnalysedMediaObserverFunc) OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error
type AnalysedMediaObservers ¶ added in v1.7.0
type AnalysedMediaObservers []AnalysedMediaObserver
func (AnalysedMediaObservers) OnAnalysedMedia ¶ added in v1.7.0
func (a AnalysedMediaObservers) OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error
type Analyser ¶ added in v1.6.26
type Analyser interface {
Analyse(ctx context.Context, found FoundMedia) (*AnalysedMedia, error)
}
Analyser is a method to get the details of the media from the content of the file.
type AnalyserDecorator ¶ added in v1.5.0
type AnalyserDecorator interface {
Decorate(analyse Analyser, observers ...AnalyserDecoratorObserver) Analyser
}
AnalyserDecorator allows to customize the Analyser behaviour, like adding a cache.
type AnalyserDecoratorObserver ¶ added in v1.6.26
type AnalyserDecoratorObserver interface {
OnSkipDelegateAnalyser(ctx context.Context, found FoundMedia) error
}
AnalyserDecoratorObserver is used to observe the decorator (if the cache hits, it will call this observer).
type AnalyserFromMediaDetails ¶ added in v1.7.0
type AnalyserFromMediaDetails struct {
DetailsReaders []DetailsReader
// contains filtered or unexported fields
}
AnalyserFromMediaDetails is using DetailsReader to extract data from the file (EXIF, MP4, ...).
func (*AnalyserFromMediaDetails) Analyse ¶ added in v1.7.0
func (a *AnalyserFromMediaDetails) Analyse(ctx context.Context, found FoundMedia) (*AnalysedMedia, error)
type ArchiveMediaPort ¶ added in v1.7.0
type ArchiveMediaPort interface {
// ArchiveMedia uploads the file in the right folder but might change the name to avoid clash with other existing files. Use files name is always returned.
ArchiveMedia(owner string, media *BackingUpMediaRequest) (string, error)
}
type BackingUpMediaRequest ¶
type BackingUpMediaRequest struct {
AnalysedMedia *AnalysedMedia
CatalogReference CatalogReference
}
BackingUpMediaRequest is the requests that must be executed to back up the media
type BatchBackup ¶ added in v1.7.0
type BatchBackup struct {
CataloguerFactory CataloguerFactory
DetailsReaders []DetailsReader
InsertMediaPort InsertMediaPort
ArchivePort ArchiveMediaPort
}
func (*BatchBackup) Backup ¶ added in v1.7.0
func (b *BatchBackup) Backup(ctx context.Context, owner ownermodel.Owner, volume SourceVolume, optionsSlice ...Options) (Report, error)
Backup is analysing each media and is backing it up if not already in the catalog.
type BatchScanner ¶ added in v1.6.35
type BatchScanner struct {
CataloguerFactory CataloguerFactory
DetailsReaders []DetailsReader
}
func (*BatchScanner) Scan ¶ added in v1.6.35
func (s *BatchScanner) Scan(ctx context.Context, owner ownermodel.Owner, volume SourceVolume, optionSlice ...Options) ([]*ScannedFolder, error)
type CatalogMediaRequest ¶
type CatalogMediaRequest struct {
BackingUpMediaRequest *BackingUpMediaRequest
ArchiveFilename string // ArchiveFilename is a normalised named generated and used in archive.
}
CatalogMediaRequest is the request passed to Archive domain
type CatalogReference ¶ added in v1.5.54
type CatalogReference interface {
// Exists returns true if the media exists in the catalog
Exists() bool
// AlbumCreated returns true if the album was created during the cataloger process
AlbumCreated() bool
// AlbumFolderName returns the name of the album where the media would be stored
AlbumFolderName() string
// UniqueIdentifier is identifying the media no matter its filename, its id in the catalog (if it's in it or not), its album, ... It's its signature.
UniqueIdentifier() string
// MediaId is the id of the media in the catalog and in the archive
MediaId() string
}
CatalogReference is used to project where a media will fit in the catalog: its ID and its album.
type CatalogReferencerObserver ¶ added in v1.6.32
type CatalogReferencerObserver interface {
OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error
}
type CatalogReferencerObserverFunc ¶ added in v1.6.35
type CatalogReferencerObserverFunc func(ctx context.Context, requests []BackingUpMediaRequest) error
func (CatalogReferencerObserverFunc) OnMediaCatalogued ¶ added in v1.6.35
func (c CatalogReferencerObserverFunc) OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error
type CatalogReferencerObservers ¶ added in v1.7.0
type CatalogReferencerObservers []CatalogReferencerObserver
func (CatalogReferencerObservers) OnMediaCatalogued ¶ added in v1.7.0
func (c CatalogReferencerObservers) OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error
type Cataloguer ¶ added in v1.6.32
type Cataloguer interface {
Reference(ctx context.Context, medias []*AnalysedMedia, observer CatalogReferencerObserver) error
}
type CataloguerFactory ¶ added in v1.6.35
type CataloguerFactory interface {
NewOwnerScopedCataloguer(ctx context.Context, owner ownermodel.Owner) (Cataloguer, error)
}
CataloguerFactory returns a Cataloguer scoped to a single owner. Implementations can be read-only, or have the behaviour of creating missing albums.
type CataloguerFilterObserver ¶ added in v1.6.32
type CataloguerFilterObserver interface {
OnFilteredOut(ctx context.Context, media AnalysedMedia, reference CatalogReference, cause error) error
}
type CataloguerFilterObservers ¶ added in v1.7.0
type CataloguerFilterObservers []CataloguerFilterObserver
func (CataloguerFilterObservers) OnFilteredOut ¶ added in v1.7.0
func (c CataloguerFilterObservers) OnFilteredOut(ctx context.Context, media AnalysedMedia, reference CatalogReference, cause error) error
type ClosableFoundMedia ¶
type ClosableFoundMedia interface {
Close() error
}
ClosableFoundMedia can be implemented alongside FoundMedia if the implementation requires to release resources once the media has been handled.
type ConcurrencyParameters ¶ added in v1.6.35
type ConcurrencyParameters struct {
ConcurrentAnalyserRoutines int // ConcurrentAnalyserRoutines is the number of concurrent analyser (read files, compute hash, filter out duplicates, ...)
ConcurrentCataloguerRoutines int // ConcurrentCataloguerRoutines is the number of concurrent cataloguer (find album, create new albums)
ConcurrentUploaderRoutines int // ConcurrentUploaderRoutines is the number of concurrent uploader (upload files)
}
func (ConcurrencyParameters) NumberOfConcurrentAnalyserRoutines ¶ added in v1.6.35
func (c ConcurrencyParameters) NumberOfConcurrentAnalyserRoutines() int
func (ConcurrencyParameters) NumberOfConcurrentCataloguerRoutines ¶ added in v1.6.35
func (c ConcurrencyParameters) NumberOfConcurrentCataloguerRoutines() int
func (ConcurrencyParameters) NumberOfConcurrentUploaderRoutines ¶ added in v1.6.35
func (c ConcurrencyParameters) NumberOfConcurrentUploaderRoutines() int
type DetailsReader ¶ added in v1.7.0
type DetailsReader interface {
// Supports returns true if the file can be parsed with this reader. False otherwise.
Supports(media FoundMedia, mediaType MediaType) bool
// ReadDetails extracts metadata from the content of the file.
ReadDetails(reader io.Reader, options DetailsReaderOptions) (*MediaDetails, error)
}
type DetailsReaderOptions ¶
type DetailsReaderOptions struct {
Fast bool // Fast true indicate the parser should focus at extracting the date, nothing else TODO can be retired
}
type ExtraCounts ¶ added in v1.6.27
type ExtraCounts struct {
Cached MediaCounter
Rejected MediaCounter
}
func (ExtraCounts) String ¶ added in v1.6.27
func (c ExtraCounts) String() interface{}
type FoundMedia ¶
type FoundMedia interface {
// MediaPath return breakdown of the absolute path of the media.
MediaPath() MediaPath
// ReadMedia reads content of the file ; it might not be optimised to call it several times (see VolumeToBackup)
ReadMedia() (io.ReadCloser, error)
// Size returns the size of the file
Size() int
LastModification() time.Time
String() string
}
FoundMedia represents files found on the scanned volume
func NewInMemoryMedia ¶ added in v1.5.0
func NewInMemoryMedia(name string, date time.Time, content []byte) FoundMedia
NewInMemoryMedia creates a new FoundMedia for TESTING PURPOSE ONLY
type FullMediaSignature ¶
FullMediaSignature is the business key of the media, unique per user
func (*FullMediaSignature) String ¶
func (s *FullMediaSignature) String() string
type ImageOrientation ¶
type ImageOrientation string
ImageOrientation is teh start point of stored data
type InMemoryMedia ¶ added in v1.5.0
type InMemoryMedia struct {
// contains filtered or unexported fields
}
func (*InMemoryMedia) LastModification ¶ added in v1.5.0
func (i *InMemoryMedia) LastModification() time.Time
func (*InMemoryMedia) MediaPath ¶ added in v1.5.0
func (i *InMemoryMedia) MediaPath() MediaPath
func (*InMemoryMedia) ReadMedia ¶ added in v1.5.0
func (i *InMemoryMedia) ReadMedia() (io.ReadCloser, error)
func (*InMemoryMedia) Size ¶ added in v1.5.0
func (i *InMemoryMedia) Size() int
func (*InMemoryMedia) String ¶ added in v1.5.0
func (i *InMemoryMedia) String() string
type IndexMediaPort ¶ added in v1.7.0
type IndexMediaPort interface {
// IndexMedias add to the catalog following medias
IndexMedias(owner string, requests []*CatalogMediaRequest) error
}
type InsertMediaPort ¶ added in v1.5.57
type InsertMediaPort interface {
IndexMedias(ctx context.Context, owner ownermodel.Owner, requests []*CatalogMediaRequest) error
}
type MediaCounter ¶
type MediaCounter struct {
Count int // Count is the number of medias
Size int // Size is the sum of the size of the medias
}
func NewMediaCounter ¶
func NewMediaCounter(count int, size int) MediaCounter
func (MediaCounter) Add ¶
func (c MediaCounter) Add(count int, size int) MediaCounter
Add creates a new MediaCounter with the delta applied ; initial MediaCounter is not updated.
func (MediaCounter) AddCounter ¶
func (c MediaCounter) AddCounter(counter MediaCounter) MediaCounter
AddCounter creates a new MediaCounter which is the sum of the 2 counters provided.
func (MediaCounter) IsZero ¶
func (c MediaCounter) IsZero() bool
IsZero returns true if it's the default value
type MediaDetails ¶
type MediaDetails struct {
Width, Height int
DateTime time.Time
Orientation ImageOrientation
Make string
Model string
GPSLatitude, GPSLongitude float64
Duration int64 // Duration is the length, in milliseconds, of a video
VideoEncoding string // VideoEncoding is the codec used to encode the video (ex: 'H264')
}
func (*MediaDetails) String ¶
func (s *MediaDetails) String() string
type MediaPath ¶
type MediaPath struct {
ParentFullPath string // ParentFullPath is the absolute path of the media folder (URL = ParentFullPath + Filename)
Root string // Root is the path or URL representing the volume in which the media has been found. (URL = Root + Path + Filename)
Path string // Path is the directory path relative to Root: URL = Root + Path + Filename
Filename string // Filename does not contain any slash, and contains the extension
ParentDir string // ParentDir is the name of the directory: dirname(Root + Path)
}
MediaPath is a breakdown of an absolute path, or URL, agnostic of its origin.
type NopeAnalyserDecorator ¶ added in v1.5.0
type NopeAnalyserDecorator struct {
}
NopeAnalyserDecorator is a default implementation for AnalyserDecorator which doesn't decorate the AnalyseMediaFunc.
func (*NopeAnalyserDecorator) Decorate ¶ added in v1.5.0
func (n *NopeAnalyserDecorator) Decorate(analyseFunc Analyser, observers ...AnalyserDecoratorObserver) Analyser
type Options ¶
type Options struct {
RestrictedAlbumFolderName map[string]interface{} // RestrictedAlbumFolderName will restrict the media to only back up medias that are in one of these albums
Listener interface{} // Listener will receive progress events.
SkipRejects bool // SkipRejects mode will report any analysis error, or missing timestamp, and continue.
AnalyserDecorator AnalyserDecorator // AnalyserDecorator is an optional decorator to add concept like caching (might be nil)
ConcurrencyParameters ConcurrencyParameters
BatchSize int // BatchSize is the number of items to read from the database at once (used by analyser) ; default to the maximum DynamoDB can handle
RejectDir string // RejectDir is the directory where rejected files will be copied
ChannelSize int // ChannelSize is a hint of the size of the channels to use. Default is set in the `chain` package (255).
}
func OptionsAnalyserDecorator ¶ added in v1.7.0
func OptionsAnalyserDecorator(analyserDecorator AnalyserDecorator) Options
OptionsAnalyserDecorator adds a decorator on analysis function ; argument can be nil. Used to add a cache.
func OptionsBatchSize ¶ added in v1.6.35
func OptionsChannelSize ¶ added in v1.7.1
func OptionsConcurrentAnalyserRoutines ¶ added in v1.6.35
func OptionsConcurrentCataloguerRoutines ¶ added in v1.6.35
func OptionsConcurrentUploaderRoutines ¶ added in v1.6.35
func OptionsOnlyAlbums ¶ added in v1.7.0
OptionsOnlyAlbums restricts backed up medias to those in these albums
func OptionsSkipRejects ¶ added in v1.6.35
OptionsSkipRejects disables the strict mode and ignores invalid files (wrong / no date, ...)
func OptionsWithListener ¶ added in v1.7.0
func OptionsWithListener(listener interface{}) Options
OptionsWithListener adds a listener tracking the progress of the scan/backup
func OptionsWithRejectDir ¶ added in v1.7.0
func ReduceOptions ¶ added in v1.6.27
func (Options) GetAnalyserDecorator ¶ added in v1.5.0
func (o Options) GetAnalyserDecorator() AnalyserDecorator
GetAnalyserDecorator is returning the AnalyserDecorator or NopeAnalyserDecorator, never nil.
func (Options) GetBatchSize ¶ added in v1.6.35
type PostAnalyseFilter ¶
type PostAnalyseFilter interface {
// AcceptAnalysedMedia returns TRUE if the media should be backed-up.
AcceptAnalysedMedia(media *AnalysedMedia, folderName string) bool
}
type RejectedMediaObserver ¶ added in v1.6.26
type RejectedMediaObserver interface {
// OnRejectedMedia is called when the media is invalid and cannot be used ; the error is returned only if there is a technical issue.
OnRejectedMedia(ctx context.Context, found FoundMedia, cause error) error
}
type RejectedMediaObservers ¶ added in v1.7.0
type RejectedMediaObservers []RejectedMediaObserver
func (RejectedMediaObservers) OnRejectedMedia ¶ added in v1.7.0
func (a RejectedMediaObservers) OnRejectedMedia(ctx context.Context, found FoundMedia, cause error) error
type Report ¶ added in v1.7.0
type Report interface {
Skipped() MediaCounter
CountPerAlbum() map[string]*AlbumReport
}
type ScannedFolder ¶
type ScannedFolder struct {
Name string
RelativePath string // RelativePath can be used for display purpose
FolderName string // FolderName is the original folder name (Name with date prefix that have been removed)
AbsolutePath string // AbsolutePath is used to create a new SourceVolume
Start, End time.Time // Start and End are the beginning of the day of the first media, and the beginning of the day following the last media.
Distribution map[string]MediaCounter // Distribution is the number of media found for each day (format YYYY-MM-DD)
RejectsCount int // RejectsCount is the number of media that have been rejected or filtered out in the folder
}
ScannedFolder represents a (sub)folder in the scanned target
func (*ScannedFolder) PushBoundaries ¶
func (f *ScannedFolder) PushBoundaries(date time.Time, size int)
PushBoundaries is updating the ScannedFolder dates, and update the counter.
type SourceVolume ¶
type SourceVolume interface {
String() string
FindMedias(ctx context.Context) ([]FoundMedia, error)
}
type TimelinePort ¶ added in v1.7.0
type TrackAnalysed ¶
type TrackAnalysed interface {
OnAnalysed(done, total MediaCounter, others ExtraCounts)
}
type TrackEvents ¶
type TrackEvents interface {
OnEvent(event progressEvent)
}
type TrackScanComplete ¶
type TrackScanComplete interface {
OnScanComplete(total MediaCounter)
}
type TrackUploaded ¶
type TrackUploaded interface {
OnUploaded(done, total MediaCounter)
}
TrackUploaded includes both uploaded and skipped
Source Files
¶
- adapters.go
- backup.go
- backup_copy_rejects.go
- backup_report.go
- backup_uploader.go
- domain_analyser.go
- domain_analyser_aggregate.go
- domain_cataloguer.go
- domain_cataloguer_filters.go
- foundmedia_inmemory.go
- model_media.go
- model_options.go
- multithreaded_chain.go
- runner_executor_scan.go
- scan.go
- scan_logger.go
- scan_report.go
- tracker_v2.go