vfs

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 6, 2020 License: GPL-3.0 Imports: 29 Imported by: 0

Documentation

Overview

Package vfs provides local and remote filesystems support

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetSFTPError 

func GetSFTPError(fs Fs, err error) error

GetSFTPError returns an sftp error from a filesystem error

func IsDirectory 

func IsDirectory(fs Fs, path string) (bool, error)

IsDirectory checks if a path exists and is a directory

func IsLocalOsFs 

func IsLocalOsFs(fs Fs) bool

IsLocalOsFs returns true if fs is the local filesystem implementation

func SetPathPermissions 

func SetPathPermissions(fs Fs, path string, uid int, gid int)

SetPathPermissions calls fs.Chown. It does nothing for local filesystem on windows

func ValidateGCSFsConfig 

func ValidateGCSFsConfig(config *GCSFsConfig, credentialsFilePath string) error

ValidateGCSFsConfig returns nil if the specified GCS config is valid, otherwise an error

func ValidateS3FsConfig 

func ValidateS3FsConfig(config *S3FsConfig) error

ValidateS3FsConfig returns nil if the specified s3 config is valid, otherwise an error

Types

type BaseVirtualFolder 

type BaseVirtualFolder struct {
	ID            int64  `json:"id"`
	MappedPath    string `json:"mapped_path"`
	UsedQuotaSize int64  `json:"used_quota_size"`
	// Used quota as number of files
	UsedQuotaFiles int `json:"used_quota_files"`
	// Last quota update as unix timestamp in milliseconds
	LastQuotaUpdate int64 `json:"last_quota_update"`
	// list of usernames associated with this virtual folder
	Users []string `json:"users,omitempty"`
}

BaseVirtualFolder defines the path for the virtual folder and the used quota limits. The same folder can be shared among multiple users and each user can have different quota limits or a different virtual path.

func (*BaseVirtualFolder) GetQuotaSummary 

func (v *BaseVirtualFolder) GetQuotaSummary() string

GetQuotaSummary returns used quota and last update as string

func (*BaseVirtualFolder) GetUsersAsString 

func (v *BaseVirtualFolder) GetUsersAsString() string

GetUsersAsString returns the list of users as comma separated string

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo implements os.FileInfo for a file in S3.

func NewFileInfo 

func NewFileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) FileInfo

NewFileInfo creates file info.

func (FileInfo) IsDir 

func (fi FileInfo) IsDir() bool

IsDir provides the abbreviation for Mode().IsDir()

func (FileInfo) ModTime 

func (fi FileInfo) ModTime() time.Time

ModTime provides the last modification time.

func (FileInfo) Mode 

func (fi FileInfo) Mode() os.FileMode

Mode provides the file mode bits

func (FileInfo) Name 

func (fi FileInfo) Name() string

Name provides the base name of the file.

func (FileInfo) Size 

func (fi FileInfo) Size() int64

Size provides the length in bytes for a file.

func (FileInfo) Sys 

func (fi FileInfo) Sys() interface{}

Sys provides the underlying data source (can return nil)

type Fs 

type Fs interface {
	Name() string
	ConnectionID() string
	Stat(name string) (os.FileInfo, error)
	Lstat(name string) (os.FileInfo, error)
	Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error)
	Create(name string, flag int) (*os.File, *PipeWriter, func(), error)
	Rename(source, target string) error
	Remove(name string, isDir bool) error
	Mkdir(name string) error
	Symlink(source, target string) error
	Chown(name string, uid int, gid int) error
	Chmod(name string, mode os.FileMode) error
	Chtimes(name string, atime, mtime time.Time) error
	ReadDir(dirname string) ([]os.FileInfo, error)
	IsUploadResumeSupported() bool
	IsAtomicUploadSupported() bool
	CheckRootPath(username string, uid int, gid int) bool
	ResolvePath(sftpPath string) (string, error)
	IsNotExist(err error) bool
	IsPermission(err error) bool
	ScanRootDirContents() (int, int64, error)
	GetDirSize(dirname string) (int, int64, error)
	GetAtomicUploadPath(name string) string
	GetRelativePath(name string) string
	Walk(root string, walkFn filepath.WalkFunc) error
	Join(elem ...string) string
}

Fs defines the interface for filesystem backends

func NewGCSFs 

func NewGCSFs(connectionID, localTempDir string, config GCSFsConfig) (Fs, error)

NewGCSFs returns an GCSFs object that allows to interact with Google Cloud Storage

func NewOsFs 

func NewOsFs(connectionID, rootDir string, virtualFolders []VirtualFolder) Fs

NewOsFs returns an OsFs object that allows to interact with local Os filesystem

func NewS3Fs 

func NewS3Fs(connectionID, localTempDir string, config S3FsConfig) (Fs, error)

NewS3Fs returns an S3Fs object that allows to interact with an s3 compatible object storage

type GCSFs 

type GCSFs struct {
	// contains filtered or unexported fields
}

GCSFs is a Fs implementation for Google Cloud Storage.

func (GCSFs) CheckRootPath 

func (fs GCSFs) CheckRootPath(username string, uid int, gid int) bool

CheckRootPath creates the specified root directory if it does not exists

func (GCSFs) Chmod 

func (GCSFs) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of the named file to mode. Silently ignored.

func (GCSFs) Chown 

func (GCSFs) Chown(name string, uid int, gid int) error

Chown changes the numeric uid and gid of the named file. Silently ignored.

func (GCSFs) Chtimes 

func (GCSFs) Chtimes(name string, atime, mtime time.Time) error

Chtimes changes the access and modification times of the named file. Silently ignored.

func (GCSFs) ConnectionID 

func (fs GCSFs) ConnectionID() string

ConnectionID returns the SSH connection ID associated to this Fs implementation

func (GCSFs) Create 

func (fs GCSFs) Create(name string, flag int) (*os.File, *PipeWriter, func(), error)

Create creates or opens the named file for writing

func (GCSFs) GetAtomicUploadPath 

func (GCSFs) GetAtomicUploadPath(name string) string

GetAtomicUploadPath returns the path to use for an atomic upload. S3 uploads are already atomic, we never call this method for S3

func (GCSFs) GetDirSize 

func (GCSFs) GetDirSize(dirname string) (int, int64, error)

GetDirSize returns the number of files and the size for a folder including any subfolders

func (GCSFs) GetRelativePath 

func (fs GCSFs) GetRelativePath(name string) string

GetRelativePath returns the path for a file relative to the user's home dir. This is the path as seen by SFTP users

func (GCSFs) IsAtomicUploadSupported 

func (GCSFs) IsAtomicUploadSupported() bool

IsAtomicUploadSupported returns true if atomic upload is supported. S3 uploads are already atomic, we don't need to upload to a temporary file

func (GCSFs) IsNotExist 

func (GCSFs) IsNotExist(err error) bool

IsNotExist returns a boolean indicating whether the error is known to report that a file or directory does not exist

func (GCSFs) IsPermission 

func (GCSFs) IsPermission(err error) bool

IsPermission returns a boolean indicating whether the error is known to report that permission is denied.

func (GCSFs) IsUploadResumeSupported 

func (GCSFs) IsUploadResumeSupported() bool

IsUploadResumeSupported returns true if upload resume is supported. SFTP Resume is not supported on S3

func (GCSFs) Join 

func (GCSFs) Join(elem ...string) string

Join joins any number of path elements into a single path

func (GCSFs) Lstat 

func (fs GCSFs) Lstat(name string) (os.FileInfo, error)

Lstat returns a FileInfo describing the named file

func (GCSFs) Mkdir 

func (fs GCSFs) Mkdir(name string) error

Mkdir creates a new directory with the specified name and default permissions

func (GCSFs) Name 

func (fs GCSFs) Name() string

Name returns the name for the Fs implementation

func (GCSFs) Open 

func (fs GCSFs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error)

Open opens the named file for reading

func (GCSFs) ReadDir 

func (fs GCSFs) ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of directory entries.

func (GCSFs) Remove 

func (fs GCSFs) Remove(name string, isDir bool) error

Remove removes the named file or (empty) directory.

func (GCSFs) Rename 

func (fs GCSFs) Rename(source, target string) error

Rename renames (moves) source to target. We don't support renaming non empty directories since we should rename all the contents too and this could take long time: think about directories with thousands of files, for each file we should execute a CopyObject call.

func (GCSFs) ResolvePath 

func (fs GCSFs) ResolvePath(sftpPath string) (string, error)

ResolvePath returns the matching filesystem path for the specified sftp path

func (GCSFs) ScanRootDirContents 

func (fs GCSFs) ScanRootDirContents() (int, int64, error)

ScanRootDirContents returns the number of files contained in the bucket, and their size

func (GCSFs) Stat 

func (fs GCSFs) Stat(name string) (os.FileInfo, error)

Stat returns a FileInfo describing the named file 

func (GCSFs) Symlink(source, target string) error

Symlink creates source as a symbolic link to target.

func (GCSFs) Walk 

func (GCSFs) Walk(root string, walkFn filepath.WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root

type GCSFsConfig 

type GCSFsConfig struct {
	Bucket string `json:"bucket,omitempty"`
	// KeyPrefix is similar to a chroot directory for local filesystem.
	// If specified then the SFTP user will only see objects that starts
	// with this prefix and so you can restrict access to a specific
	// folder. The prefix, if not empty, must not start with "/" and must
	// end with "/".
	// If empty the whole bucket contents will be available
	KeyPrefix            string `json:"key_prefix,omitempty"`
	CredentialFile       string `json:"-"`
	Credentials          string `json:"credentials,omitempty"`
	AutomaticCredentials int    `json:"automatic_credentials,omitempty"`
	StorageClass         string `json:"storage_class,omitempty"`
}

GCSFsConfig defines the configuration for Google Cloud Storage based filesystem

type OsFs 

type OsFs struct {
	// contains filtered or unexported fields
}

OsFs is a Fs implementation that uses functions provided by the os package.

func (OsFs) CheckRootPath 

func (fs OsFs) CheckRootPath(username string, uid int, gid int) bool

CheckRootPath creates the root directory if it does not exists

func (OsFs) Chmod 

func (OsFs) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of the named file to mode

func (OsFs) Chown 

func (OsFs) Chown(name string, uid int, gid int) error

Chown changes the numeric uid and gid of the named file.

func (OsFs) Chtimes 

func (OsFs) Chtimes(name string, atime, mtime time.Time) error

Chtimes changes the access and modification times of the named file

func (OsFs) ConnectionID 

func (fs OsFs) ConnectionID() string

ConnectionID returns the SSH connection ID associated to this Fs implementation

func (OsFs) Create 

func (OsFs) Create(name string, flag int) (*os.File, *PipeWriter, func(), error)

Create creates or opens the named file for writing

func (OsFs) GetAtomicUploadPath 

func (OsFs) GetAtomicUploadPath(name string) string

GetAtomicUploadPath returns the path to use for an atomic upload

func (OsFs) GetDirSize 

func (fs OsFs) GetDirSize(dirname string) (int, int64, error)

GetDirSize returns the number of files and the size for a folder including any subfolders

func (*OsFs) GetFsPaths 

func (fs *OsFs) GetFsPaths(sftpPath string) (string, string)

GetFsPaths returns the base path and filesystem path for the given sftpPath. base path is the root dir or matching the virtual folder dir for the sftpPath. file path is the filesystem path matching the sftpPath

func (OsFs) GetRelativePath 

func (fs OsFs) GetRelativePath(name string) string

GetRelativePath returns the path for a file relative to the user's home dir. This is the path as seen by SFTP users

func (OsFs) IsAtomicUploadSupported 

func (OsFs) IsAtomicUploadSupported() bool

IsAtomicUploadSupported returns true if atomic upload is supported

func (OsFs) IsNotExist 

func (OsFs) IsNotExist(err error) bool

IsNotExist returns a boolean indicating whether the error is known to report that a file or directory does not exist

func (OsFs) IsPermission 

func (OsFs) IsPermission(err error) bool

IsPermission returns a boolean indicating whether the error is known to report that permission is denied.

func (OsFs) IsUploadResumeSupported 

func (OsFs) IsUploadResumeSupported() bool

IsUploadResumeSupported returns true if upload resume is supported

func (OsFs) Join 

func (OsFs) Join(elem ...string) string

Join joins any number of path elements into a single path

func (OsFs) Lstat 

func (OsFs) Lstat(name string) (os.FileInfo, error)

Lstat returns a FileInfo describing the named file

func (OsFs) Mkdir 

func (OsFs) Mkdir(name string) error

Mkdir creates a new directory with the specified name and default permissions

func (OsFs) Name 

func (fs OsFs) Name() string

Name returns the name for the Fs implementation

func (OsFs) Open 

func (OsFs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error)

Open opens the named file for reading

func (OsFs) ReadDir 

func (OsFs) ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of directory entries.

func (OsFs) Remove 

func (OsFs) Remove(name string, isDir bool) error

Remove removes the named file or (empty) directory.

func (OsFs) Rename 

func (OsFs) Rename(source, target string) error

Rename renames (moves) source to target

func (OsFs) ResolvePath 

func (fs OsFs) ResolvePath(sftpPath string) (string, error)

ResolvePath returns the matching filesystem path for the specified sftp path

func (OsFs) ScanRootDirContents 

func (fs OsFs) ScanRootDirContents() (int, int64, error)

ScanRootDirContents returns the number of files contained in a directory and their size

func (OsFs) Stat 

func (OsFs) Stat(name string) (os.FileInfo, error)

Stat returns a FileInfo describing the named file 

func (OsFs) Symlink(source, target string) error

Symlink creates source as a symbolic link to target.

func (OsFs) Walk 

func (OsFs) Walk(root string, walkFn filepath.WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root

type PipeWriter 

type PipeWriter struct {
	// contains filtered or unexported fields
}

PipeWriter defines a wrapper for pipeat.PipeWriterAt.

func NewPipeWriter 

func NewPipeWriter(w *pipeat.PipeWriterAt) *PipeWriter

NewPipeWriter initializes a new PipeWriter

func (*PipeWriter) Close 

func (p *PipeWriter) Close() error

Close waits for the upload to end, closes the pipeat.PipeWriterAt and returns an error if any.

func (*PipeWriter) Done 

func (p *PipeWriter) Done(err error)

Done unlocks other goroutines waiting on Close(). It must be called when the upload ends

func (*PipeWriter) WriteAt 

func (p *PipeWriter) WriteAt(data []byte, off int64) (int, error)

WriteAt is a wrapper for pipeat WriteAt

type QuotaCheckResult 

type QuotaCheckResult struct {
	HasSpace     bool
	AllowedSize  int64
	AllowedFiles int
	UsedSize     int64
	UsedFiles    int
	QuotaSize    int64
	QuotaFiles   int
}

QuotaCheckResult defines the result for a quota check

func (*QuotaCheckResult) GetRemainingFiles 

func (q *QuotaCheckResult) GetRemainingFiles() int

GetRemainingFiles returns the remaining allowed files

func (*QuotaCheckResult) GetRemainingSize 

func (q *QuotaCheckResult) GetRemainingSize() int64

GetRemainingSize returns the remaining allowed size

type S3Fs 

type S3Fs struct {
	// contains filtered or unexported fields
}

S3Fs is a Fs implementation for Amazon S3 compatible object storage.

func (S3Fs) CheckRootPath 

func (fs S3Fs) CheckRootPath(username string, uid int, gid int) bool

CheckRootPath creates the specified root directory if it does not exists

func (S3Fs) Chmod 

func (S3Fs) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of the named file to mode. Silently ignored.

func (S3Fs) Chown 

func (S3Fs) Chown(name string, uid int, gid int) error

Chown changes the numeric uid and gid of the named file. Silently ignored.

func (S3Fs) Chtimes 

func (S3Fs) Chtimes(name string, atime, mtime time.Time) error

Chtimes changes the access and modification times of the named file. Silently ignored.

func (S3Fs) ConnectionID 

func (fs S3Fs) ConnectionID() string

ConnectionID returns the SSH connection ID associated to this Fs implementation

func (S3Fs) Create 

func (fs S3Fs) Create(name string, flag int) (*os.File, *PipeWriter, func(), error)

Create creates or opens the named file for writing

func (S3Fs) GetAtomicUploadPath 

func (S3Fs) GetAtomicUploadPath(name string) string

GetAtomicUploadPath returns the path to use for an atomic upload. S3 uploads are already atomic, we never call this method for S3

func (S3Fs) GetDirSize 

func (S3Fs) GetDirSize(dirname string) (int, int64, error)

GetDirSize returns the number of files and the size for a folder including any subfolders

func (S3Fs) GetRelativePath 

func (fs S3Fs) GetRelativePath(name string) string

GetRelativePath returns the path for a file relative to the user's home dir. This is the path as seen by SFTP users

func (S3Fs) IsAtomicUploadSupported 

func (S3Fs) IsAtomicUploadSupported() bool

IsAtomicUploadSupported returns true if atomic upload is supported. S3 uploads are already atomic, we don't need to upload to a temporary file

func (S3Fs) IsNotExist 

func (S3Fs) IsNotExist(err error) bool

IsNotExist returns a boolean indicating whether the error is known to report that a file or directory does not exist

func (S3Fs) IsPermission 

func (S3Fs) IsPermission(err error) bool

IsPermission returns a boolean indicating whether the error is known to report that permission is denied.

func (S3Fs) IsUploadResumeSupported 

func (S3Fs) IsUploadResumeSupported() bool

IsUploadResumeSupported returns true if upload resume is supported. SFTP Resume is not supported on S3

func (S3Fs) Join 

func (S3Fs) Join(elem ...string) string

Join joins any number of path elements into a single path

func (S3Fs) Lstat 

func (fs S3Fs) Lstat(name string) (os.FileInfo, error)

Lstat returns a FileInfo describing the named file

func (S3Fs) Mkdir 

func (fs S3Fs) Mkdir(name string) error

Mkdir creates a new directory with the specified name and default permissions

func (S3Fs) Name 

func (fs S3Fs) Name() string

Name returns the name for the Fs implementation

func (S3Fs) Open 

func (fs S3Fs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error)

Open opens the named file for reading

func (S3Fs) ReadDir 

func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of directory entries.

func (S3Fs) Remove 

func (fs S3Fs) Remove(name string, isDir bool) error

Remove removes the named file or (empty) directory.

func (S3Fs) Rename 

func (fs S3Fs) Rename(source, target string) error

Rename renames (moves) source to target. We don't support renaming non empty directories since we should rename all the contents too and this could take long time: think about directories with thousands of files, for each file we should execute a CopyObject call. TODO: rename does not work for files bigger than 5GB, implement multipart copy or wait for this pull request to be merged:

https://github.com/aws/aws-sdk-go/pull/2653

func (S3Fs) ResolvePath 

func (fs S3Fs) ResolvePath(sftpPath string) (string, error)

ResolvePath returns the matching filesystem path for the specified sftp path

func (S3Fs) ScanRootDirContents 

func (fs S3Fs) ScanRootDirContents() (int, int64, error)

ScanRootDirContents returns the number of files contained in the bucket, and their size

func (S3Fs) Stat 

func (fs S3Fs) Stat(name string) (os.FileInfo, error)

Stat returns a FileInfo describing the named file 

func (S3Fs) Symlink(source, target string) error

Symlink creates source as a symbolic link to target.

func (S3Fs) Walk 

func (S3Fs) Walk(root string, walkFn filepath.WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root

type S3FsConfig 

type S3FsConfig struct {
	Bucket string `json:"bucket,omitempty"`
	// KeyPrefix is similar to a chroot directory for local filesystem.
	// If specified then the SFTP user will only see objects that starts
	// with this prefix and so you can restrict access to a specific
	// folder. The prefix, if not empty, must not start with "/" and must
	// end with "/".
	// If empty the whole bucket contents will be available
	KeyPrefix    string `json:"key_prefix,omitempty"`
	Region       string `json:"region,omitempty"`
	AccessKey    string `json:"access_key,omitempty"`
	AccessSecret string `json:"access_secret,omitempty"`
	Endpoint     string `json:"endpoint,omitempty"`
	StorageClass string `json:"storage_class,omitempty"`
	// The buffer size (in MB) to use for multipart uploads. The minimum allowed part size is 5MB,
	// and if this value is set to zero, the default value (5MB) for the AWS SDK will be used.
	// The minimum allowed value is 5.
	// Please note that if the upload bandwidth between the SFTP client and SFTPGo is greater than
	// the upload bandwidth between SFTPGo and S3 then the SFTP client have to wait for the upload
	// of the last parts to S3 after it ends the file upload to SFTPGo, and it may time out.
	// Keep this in mind if you customize these parameters.
	UploadPartSize int64 `json:"upload_part_size,omitempty"`
	// How many parts are uploaded in parallel
	UploadConcurrency int `json:"upload_concurrency,omitempty"`
}

S3FsConfig defines the configuration for S3 based filesystem

type VirtualFolder 

type VirtualFolder struct {
	BaseVirtualFolder
	VirtualPath string `json:"virtual_path"`
	// Maximum size allowed as bytes. 0 means unlimited, -1 included in user quota
	QuotaSize int64 `json:"quota_size"`
	// Maximum number of files allowed. 0 means unlimited, -1 included in user quota
	QuotaFiles int `json:"quota_files"`
}

VirtualFolder defines a mapping between a SFTP/SCP virtual path and a filesystem path outside the user home directory. The specified paths must be absolute and the virtual path cannot be "/", it must be a sub directory. The parent directory for the specified virtual path must exist. SFTPGo will try to automatically create any missing parent directory for the configured virtual folders at user login.

func (*VirtualFolder) HasNoQuotaRestrictions 

func (v *VirtualFolder) HasNoQuotaRestrictions(checkFiles bool) bool

HasNoQuotaRestrictions returns true if no quota restrictions need to be applyed

func (*VirtualFolder) IsIncludedInUserQuota 

func (v *VirtualFolder) IsIncludedInUserQuota() bool

IsIncludedInUserQuota returns true if the virtual folder is included in user quota

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.