Documentation
¶
Overview ¶
Package azure provides functionality for interacting with Microsoft Azure services, specifically focusing on OneDrive and SharePoint storage operations through Microsoft Graph API.
The package implements various features including:
- Authentication and token management for Azure services
- Large file uploads with chunked transfer and parallel processing
- File metadata retrieval and management
- Storage quota information
- Hash verification
- Integration with rclone configuration
Core Components:
AzureClient: The main client struct that handles authentication and API operations. It manages access tokens, refresh tokens, and provides methods for file operations.
UploadParams: Configuration struct for customizing file upload behavior including chunk size, parallel processing, and retry mechanisms.
DriveQuota: Represents storage quota information including total, used, and remaining space.
Key Features:
- Automatic token refresh and management
- Parallel chunk upload with configurable workers
- Retry mechanism for failed operations
- Progress tracking and error handling
- Storage quota management
- QuickXorHash verification
Usage Example:
client, err := NewAzureClientFromRcloneConfigData(configData, "remote")
if err != nil {
log.Fatal(err)
}
params := UploadParams{
FilePath: "local/path/file.txt",
RemoteFilePath: "remote/path/file.txt",
ChunkSize: 10 * 1024 * 1024, // 10MB chunks
MaxRetries: 3,
RetryDelay: time.Second * 5,
}
fileID, err := client.Upload(httpClient, params)
The package is designed to handle large file transfers efficiently and provides robust error handling and retry mechanisms for reliable file operations.
Index ¶
- func DisplayQuotaInfo(remote string, quota *DriveQuota)
- func GetAvailableRemotes(parsedRcloneConfig *[]map[string]string) []string
- func GetRemoteConfig(parsedRcloneConfig *[]map[string]string, remoteConfig string) (map[string]string, error)
- func ParseRcloneConfigData(configData []byte) ([]map[string]string, error)
- type AzureClient
- func (client *AzureClient) EnsureTokenValid(httpClient *http.Client) error
- func (client *AzureClient) GetDriveQuota(httpClient *http.Client) (*DriveQuota, error)
- func (client *AzureClient) GetQuickXorHash(httpClient *http.Client, fileID string) (string, error)
- func (client *AzureClient) Upload(httpClient *http.Client, params UploadParams) (string, error)
- type DriveItem
- type DriveQuota
- type ProgressCallback
- type UploadParams
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DisplayQuotaInfo ¶
func DisplayQuotaInfo(remote string, quota *DriveQuota)
DisplayQuotaInfo prints quota information for a given remote drive to standard output. It displays the remote name and formatted storage values for total, used, free and trashed space.
Parameters:
- remote: string representing the remote drive name/path
- quota: pointer to DriveQuota struct containing storage quota information
The output is formatted as follows:
- Remote: <remote name>
- Total: <formatted total space>
- Used: <formatted used space>
- Free: <formatted remaining space>
- Trashed: <formatted deleted space>
func GetAvailableRemotes ¶
GetAvailableRemotes extracts and returns a slice of remote names from the parsed rclone configuration. It takes a pointer to a slice of string maps representing the parsed rclone config and iterates through each map's keys to collect all remote names.
Parameters:
- parsedRcloneConfig: A pointer to a slice of maps containing the parsed rclone configuration
Returns:
- []string: A slice containing all available remote names from the configuration
func GetRemoteConfig ¶
func GetRemoteConfig(parsedRcloneConfig *[]map[string]string, remoteConfig string) (map[string]string, error)
GetRemoteConfig retrieves the configuration map for a specified remote from parsed rclone config. It takes a pointer to a slice of string maps containing parsed rclone configurations and a remote name as input. Returns the configuration map for the specified remote if found, or an error if the remote doesn't exist.
Parameters:
- parsedRcloneConfig: Pointer to slice of maps containing parsed rclone configurations
- remoteConfig: Name of the remote configuration to retrieve
Returns:
- map[string]string: Configuration map for the specified remote
- error: Error if remote is not found or any other error occurs
func ParseRcloneConfigData ¶
ParseRcloneConfigData parses rclone configuration data from a byte slice and returns an array of configuration maps. Each configuration map represents a remote section in the rclone config, containing key-value pairs of settings.
The function processes the config data line by line, handling: - Section headers in [section-name] format - Key-value pairs in "key = value" format - Empty values in "key =" format - Blank lines and comments (lines starting with #) are ignored
Each remote section is converted into a map with its settings, including a special "remote_name" key containing the section name.
Parameters:
- configData: []byte containing the rclone configuration data
Returns:
- []map[string]string: Array of maps, each containing config settings for one remote
- error: Returns error if parsing fails
Types ¶
type AzureClient ¶
type AzureClient struct {
ClientID string
ClientSecret string
AccessToken string
RefreshToken string
Expiration time.Time
DriveID string
DriveType string
// Root folder of the remote. Sometimes a remote may not want the tool from
// uploading directly to the root folder, but instead into a custom folder.
RemoteRootFolder string
// Base url from which user can download the file.
RemoteBaseUrl string
// contains filtered or unexported fields
}
AzureClient represents a client for interacting with Microsoft Azure services. It manages authentication credentials and access tokens for Azure API operations.
Fields:
- ClientID: The application (client) ID registered in Azure Active Directory
- ClientSecret: The client secret key for authentication
- AccessToken: The current OAuth access token for API requests
- RefreshToken: Token used to obtain a new access token when expired
- Expiration: Timestamp indicating when the current access token expires
- DriveID: The identifier for the specific OneDrive instance
- DriveType: The type of drive (personal, business, sharepoint)
- mu: Mutex for handling concurrent access to client fields
func NewAzureClientFromRcloneConfigData ¶
func NewAzureClientFromRcloneConfigData(configData []byte, remoteConfig string) (*AzureClient, error)
NewAzureClientFromRcloneConfigData creates a new AzureClient instance using rclone configuration data. It takes a byte slice containing rclone config data and a remote configuration name as input.
The function parses the rclone configuration and extracts necessary Azure credentials including: - Client ID and Secret - Access and Refresh tokens - Token expiration time - Drive ID and Drive type
Parameters:
- configData: []byte containing the rclone configuration data
- remoteConfig: string specifying which remote configuration to use
Returns:
- *AzureClient: Pointer to initialized AzureClient instance
- error: Error if configuration parsing or client creation fails
func (*AzureClient) EnsureTokenValid ¶
func (client *AzureClient) EnsureTokenValid(httpClient *http.Client) error
EnsureTokenValid ensures the Azure access token is valid by checking its expiration and refreshing it if necessary. It uses a mutex to ensure thread-safe token updates.
The function performs the following steps: 1. Checks if the current token is still valid 2. If expired, requests a new token using the refresh token 3. Updates the client's access token, refresh token, and expiration time
Parameters:
- httpClient: *http.Client - The HTTP client used to make the token refresh request
Returns:
- error: Returns nil if token is valid or successfully refreshed, error otherwise
Thread-safety: This method is thread-safe as it uses a mutex to protect token updates.
func (*AzureClient) GetDriveQuota ¶
func (client *AzureClient) GetDriveQuota(httpClient *http.Client) (*DriveQuota, error)
GetDriveQuota retrieves the quota information for the user's OneDrive storage using the Microsoft Graph API. It returns a DriveQuota struct containing total storage space, used space, remaining space and deleted space in bytes.
The function automatically ensures the access token is valid before making the request. If any error occurs during the process (token validation, HTTP request, response parsing), it returns nil for DriveQuota and the corresponding error.
Parameters:
- httpClient: *http.Client - The HTTP client to use for making the request
Returns:
- *DriveQuota: Contains quota information (total, used, remaining, and deleted space)
- error: Any error encountered during the process
func (*AzureClient) GetQuickXorHash ¶
GetQuickXorHash retrieves the QuickXorHash value for a specified file from Microsoft Graph API.
Parameters:
- httpClient: *http.Client - The HTTP client used to make the request
- fileID: string - The unique identifier of the file in Microsoft OneDrive
Returns:
- string: The QuickXorHash value of the file
- error: An error object that indicates if the operation was unsuccessful
The function performs the following steps: 1. Validates the access token 2. Makes a GET request to Microsoft Graph API to fetch file metadata 3. Parses the response to extract the QuickXorHash value
Error cases:
- Invalid or expired access token
- Failed HTTP request
- Non-200 HTTP response
- Missing QuickXorHash in metadata
- JSON parsing errors
func (*AzureClient) Upload ¶
func (client *AzureClient) Upload(httpClient *http.Client, params UploadParams) (string, error)
Upload performs a large file upload to Azure storage using chunked upload with parallel processing. It creates an upload session, splits the file into chunks, and uploads them in parallel using a worker pool.
Parameters:
- httpClient: The HTTP client to use for requests
- FilePath: Local path of file to upload
- RemoteFilePath: Destination path in Azure storage
- ChunkSize: Size of each upload chunk in bytes
- ParallelChunks: Number of chunks to upload in parallel
- MaxRetries: Maximum number of retry attempts per chunk
- RetryDelay: Delay between retry attempts
Returns:
- string: The file ID of the uploaded file
- error: Any error that occurred during upload
The function implements the following features:
- Automatic token refresh
- Parallel chunk upload using worker pools
- Configurable chunk size and parallel upload count
- Retry mechanism for failed chunk uploads
- Progress tracking and error handling
type DriveItem ¶
DriveItem represents an item in a Microsoft OneDrive or SharePoint drive. It contains basic properties such as the unique identifier and name of the item.
type DriveQuota ¶
type DriveQuota struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
Remaining int64 `json:"remaining"`
Deleted int64 `json:"deleted"`
}
DriveQuota represents storage quota information for a drive. It contains details about the total storage space, used space, remaining space, and space used by items in the recycle bin.
type ProgressCallback ¶
type ProgressCallback func(uploadedBytes int64)
ProgressCallback is a function that gets called with progress updates
type UploadParams ¶
type UploadParams struct {
FilePath string
RemoteFilePath string
ChunkSize int64
MaxRetries int
RetryDelay time.Duration
AccessToken string
ProgressCallback ProgressCallback
}
UploadParams contains configuration parameters for file upload operations to Azure Blob Storage.
Fields:
- FilePath: Local path of the file to be uploaded
- RemoteFilePath: Destination path in Azure Blob Storage
- ChunkSize: Size of each upload chunk in bytes
- MaxRetries: Maximum number of retry attempts for failed uploads
- RetryDelay: Duration to wait between retry attempts
- AccessToken: Azure authentication token for the upload operation
Source Files