files

package
v0.0.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (

	// ErrAlreadyExists is returned when an upload targets an existing path with
	// overwrite disabled. It is surfaced as a sentinel so callers detect
	// "already exists" with errors.Is regardless of which protocol produced it.
	ErrAlreadyExists = errors.New("the file being created already exists")
)

Sentinel errors surfaced by the large-file upload API.

Functions

This section is empty.

Types

type AddBlockRequest 

type AddBlockRequest struct {
	Handle *int64
	Data   []byte
}

type AddBlockResponse 

type AddBlockResponse struct {
}

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewClient 

func NewClient(ctx context.Context, opts ...client.Option) (*Client, error)

func (*Client) AddBlock 

func (c *Client) AddBlock(ctx context.Context, req *AddBlockRequest, opts ...call.Option) (*AddBlockResponse, error)

Appends a block of data to the stream specified by the input handle. If the handle does not exist, this call will throw an exception with “RESOURCE_DOES_NOT_EXIST“.

If the block of data exceeds 1 MB, this call will throw an exception with “MAX_BLOCK_SIZE_EXCEEDED“.

func (*Client) Close 

func (c *Client) Close(ctx context.Context, req *CloseRequest, opts ...call.Option) (*CloseResponse, error)

Closes the stream specified by the input handle. If the handle does not exist, this call throws an exception with “RESOURCE_DOES_NOT_EXIST“.

func (*Client) Create 

func (c *Client) Create(ctx context.Context, req *CreateRequest, opts ...call.Option) (*CreateResponse, error)

Opens a stream to write to a file and returns a handle to this stream. There is a 10 minute idle timeout on this handle. If a file or directory already exists on the given path and __overwrite__ is set to false, this call will throw an exception with “RESOURCE_ALREADY_EXISTS“.

A typical workflow for file upload would be:

1. Issue a “create“ call and get a handle. 2. Issue one or more “add-block“ calls with the handle you have. 3. Issue a “close“ call with the handle you have.

func (*Client) CreateDirectory 

func (c *Client) CreateDirectory(ctx context.Context, req *CreateDirectoryRequest, opts ...call.Option) (*CreateDirectoryResponse, error)

Creates an empty directory. If necessary, also creates any parent directories of the new, empty directory (like the shell command `mkdir -p`). If called on an existing directory, returns a success response; this method is idempotent (it will succeed if the directory already exists).

func (*Client) Delete 

func (c *Client) Delete(ctx context.Context, req *DeleteRequest, opts ...call.Option) (*DeleteResponse, error)

Delete the file or directory (optionally recursively delete all files in the directory). This call throws an exception with `IO_ERROR` if the path is a non-empty directory and `recursive` is set to `false` or on other similar errors.

When you delete a large number of files, the delete operation is done in increments. The call returns a response after approximately 45 seconds with an error message (503 Service Unavailable) asking you to re-invoke the delete operation until the directory structure is fully deleted.

For operations that delete more than 10K files, we discourage using the DBFS REST API, but advise you to perform such operations in the context of a cluster, using the [File system utility (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs). `dbutils.fs` covers the functional scope of the DBFS REST API, but from notebooks. Running such operations using notebooks provides better control and manageability, such as selective deletes, and the possibility to automate periodic delete jobs.

func (*Client) DeleteDirectory 

func (c *Client) DeleteDirectory(ctx context.Context, req *DeleteDirectoryRequest, opts ...call.Option) (*DeleteDirectoryResponse, error)

Deletes an empty directory.

To delete a non-empty directory, first delete all of its contents. This can be done by listing the directory contents and deleting each file and subdirectory recursively.

func (*Client) DeleteFile 

func (c *Client) DeleteFile(ctx context.Context, req *DeleteFileRequest, opts ...call.Option) (*DeleteFileResponse, error)

Deletes a file. If the request is successful, there is no response body.

func (*Client) DownloadFile 

func (c *Client) DownloadFile(ctx context.Context, req *DownloadFileRequest, opts ...call.Option) (*DownloadFileResponse, error)

Downloads a file. The file contents are the response body. This is a standard HTTP file download, not a JSON RPC. It supports the Range and If-Unmodified-Since HTTP headers.

func (*Client) GetDirectoryMetadata 

func (c *Client) GetDirectoryMetadata(ctx context.Context, req *GetDirectoryMetadataRequest, opts ...call.Option) (*GetDirectoryMetadataResponse, error)

Get the metadata of a directory. The response HTTP headers contain the metadata. There is no response body.

This method is useful to check if a directory exists and the caller has access to it.

If you wish to ensure the directory exists, you can instead use `PUT`, which will create the directory if it does not exist, and is idempotent (it will succeed if the directory already exists).

func (*Client) GetFileMetadata 

func (c *Client) GetFileMetadata(ctx context.Context, req *GetFileMetadataRequest, opts ...call.Option) (*GetFileMetadataResponse, error)

Get the metadata of a file. The response HTTP headers contain the metadata. There is no response body.

func (*Client) GetStatus 

func (c *Client) GetStatus(ctx context.Context, req *GetStatusRequest, opts ...call.Option) (*GetStatusResponse, error)

Gets the file information for a file or directory. If the file or directory does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`.

func (*Client) List 

func (c *Client) List(ctx context.Context, req *ListStatusRequest, opts ...call.Option) (*ListStatusResponse, error)

List the contents of a directory, or details of the file. If the file or directory does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`.

When calling list on a large directory, the list operation will time out after approximately 60 seconds. We strongly recommend using list only on directories containing less than 10K files and discourage using the DBFS REST API for operations that list more than 10K files. Instead, we recommend that you perform such operations in the context of a cluster, using the [File system utility (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs), which provides the same functionality without timing out.

func (*Client) ListDirectoryContents 

func (c *Client) ListDirectoryContents(ctx context.Context, req *ListDirectoryContentsRequest, opts ...call.Option) (*ListDirectoryResponse, error)

Returns the contents of a directory. If there is no directory at the specified path, the API returns an HTTP 404 error.

func (*Client) ListDirectoryContentsIter 

func (c *Client) ListDirectoryContentsIter(ctx context.Context, req *ListDirectoryContentsRequest, opts ...call.Option) iter.Seq2[*DirectoryEntry, error]

ListDirectoryContentsIter returns an iterator that iterates over the results of ListDirectoryContents.

For example: 

for item, err := range c.ListDirectoryContentsIter(ctx, &ListDirectoryContentsRequest{}) {
  if err != nil {
    return err
  }
  fmt.Println(item)
}

Options opts are passed to each ListDirectoryContents call made by the iterator under the hood.

Callers who need custom pagination logic should use ListDirectoryContents directly.

func (*Client) Mkdirs 

func (c *Client) Mkdirs(ctx context.Context, req *MkDirsRequest, opts ...call.Option) (*MkDirsResponse, error)

Creates the given directory and necessary parent directories if they do not exist. If a file (not a directory) exists at any prefix of the input path, this call throws an exception with `RESOURCE_ALREADY_EXISTS`. **Note**: If this operation fails, it might have succeeded in creating some of the necessary parent directories.

func (*Client) Move 

func (c *Client) Move(ctx context.Context, req *MoveRequest, opts ...call.Option) (*MoveResponse, error)

Moves a file from one location to another location within DBFS. If the source file does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. If a file already exists in the destination path, this call throws an exception with `RESOURCE_ALREADY_EXISTS`. If the given source path is a directory, this call always recursively moves all files.

func (*Client) Put 

func (c *Client) Put(ctx context.Context, req *PutRequest, opts ...call.Option) (*PutResponse, error)

Uploads a file through the use of multipart form post. It is mainly used for streaming uploads, but can also be used as a convenient single call for data upload.

Alternatively you can pass contents as base64 string.

The amount of data that can be passed (when not streaming) using the __contents__ parameter is limited to 1 MB. `MAX_BLOCK_SIZE_EXCEEDED` will be thrown if this limit is exceeded.

If you want to upload large files, use the streaming upload. For details, see :method:dbfs/create, :method:dbfs/addBlock, :method:dbfs/close.

func (*Client) Read 

func (c *Client) Read(ctx context.Context, req *ReadRequest, opts ...call.Option) (*ReadResponse, error)

Returns the contents of a file. If the file does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. If the path is a directory, the read length is negative, or if the offset is negative, this call throws an exception with `INVALID_PARAMETER_VALUE`. If the read length exceeds 1 MB, this call throws an exception with `MAX_READ_SIZE_EXCEEDED`.

If `offset + length` exceeds the number of bytes in a file, it reads the contents until the end of file.

func (*Client) Upload 

func (c *Client) Upload(ctx context.Context, filePath string, contents io.Reader, opts ...UploadOption) (*UploadResult, error)

Upload uploads contents to the remote file at filePath, which must be an absolute path such as "/Volumes/catalog/schema/volume/file". It chooses a single-shot, multipart, or resumable upload based on the stream size (when the reader is seekable) and the protocol the workspace's Files API selects.

When contents implements io.ReaderAt (for example an *os.File or a *bytes.Reader), parts are read from it with concurrent positioned reads rather than buffered in memory. A reader without random access has each in-flight part buffered as it is read.

ctx bounds the entire operation, including all retries; pass a context with a deadline to cap the total upload time.

func (*Client) UploadFile 

func (c *Client) UploadFile(ctx context.Context, req *UploadFileRequest, opts ...call.Option) (*UploadFileResponse, error)

Uploads a file of up to 5 GiB. The file contents should be sent as the request body as raw bytes (an octet stream); do not encode or otherwise modify the bytes before sending. The contents of the resulting file will be exactly the bytes sent in the request body. If the request is successful, there is no response body.

func (*Client) UploadFrom 

func (c *Client) UploadFrom(ctx context.Context, filePath, sourcePath string, opts ...UploadOption) (*UploadResult, error)

UploadFrom uploads the local file at sourcePath to the remote file at filePath. The local file is opened as needed; its size is always known, so it never takes the non-seekable path.

Parts are read from the file on demand rather than buffered, so the upload covers the file as of its size when the call begins: bytes appended afterward are not included, and truncating the file mid-upload fails the upload (it does not produce a partial object). Do not modify the file while an upload is in progress.

ctx bounds the entire operation, including all retries; pass a context with a deadline to cap the total upload time.

type CloseRequest 

type CloseRequest struct {
	Handle *int64
}

type CloseResponse 

type CloseResponse struct {
}

type CreateDirectoryRequest 

type CreateDirectoryRequest struct {
	DirectoryPath *string
}

Create a directory.

type CreateDirectoryResponse 

type CreateDirectoryResponse struct {
}

type CreateRequest 

type CreateRequest struct {
	Path      *string
	Overwrite *bool
}

type CreateResponse 

type CreateResponse struct {
	Handle *int64
}

type DeleteDirectoryRequest 

type DeleteDirectoryRequest struct {
	DirectoryPath *string
}

Delete a directory.

type DeleteDirectoryResponse 

type DeleteDirectoryResponse struct {
}

type DeleteFileRequest 

type DeleteFileRequest struct {
	FilePath *string
}

Delete a file.

type DeleteFileResponse 

type DeleteFileResponse struct {
}

type DeleteRequest 

type DeleteRequest struct {
	Path      *string
	Recursive *bool
}

type DeleteResponse 

type DeleteResponse struct {
}

type DirectoryEntry 

type DirectoryEntry struct {
	FileSize     *int
	IsDirectory  *bool
	LastModified *int
	Name         *string
	Path         *string
}

type DownloadFileRequest 

type DownloadFileRequest struct {
	FilePath          *string
	Range             *string
	IfUnmodifiedSince *string
}

Download a file.

type DownloadFileResponse 

type DownloadFileResponse struct {
	ContentLength *int64
	ContentType   *string
	Contents      io.ReadCloser
	LastModified  *string
}

type FileInfo 

type FileInfo struct {
	Path             *string
	IsDir            *bool
	FileSize         *int64
	ModificationTime *int64
}

Stores the attributes of a file or directory..

type GetDirectoryMetadataRequest 

type GetDirectoryMetadataRequest struct {
	DirectoryPath *string
}

Get directory metadata.

type GetDirectoryMetadataResponse 

type GetDirectoryMetadataResponse struct {
}

type GetFileMetadataRequest 

type GetFileMetadataRequest struct {
	FilePath          *string
	Range             *string
	IfUnmodifiedSince *string
}

Get file metadata.

type GetFileMetadataResponse 

type GetFileMetadataResponse struct {
	ContentLength *int64
	ContentType   *string
	LastModified  *string
}

type GetStatusRequest 

type GetStatusRequest struct {
	Path *string
}

type GetStatusResponse 

type GetStatusResponse struct {
	Path             *string
	IsDir            *bool
	FileSize         *int64
	ModificationTime *int64
}

type Limiter 

type Limiter interface {
	// Acquire blocks until a slot is free or ctx is cancelled; on cancellation it
	// returns ctx.Err() and the caller must not Release.
	Acquire(ctx context.Context) error
	// Release returns a slot taken by a successful Acquire.
	Release()
}

Limiter bounds the number of concurrent cloud-leg transfers an upload may run. The engine acquires one unit before each transfer (every multipart part PUT, the single-shot PUT, and every resumable chunk) and releases it when that transfer returns. Pass the same Limiter to multiple Upload calls -- for example when copying many files at once -- to cap their combined concurrency. Implementations must be safe for concurrent use.

func NewLimiter 

func NewLimiter(n int) Limiter

NewLimiter returns a Limiter permitting at most n concurrent transfers. A value of n <= 0 yields an unlimited limiter whose Acquire never blocks.

type ListDirectoryContentsRequest 

type ListDirectoryContentsRequest struct {
	DirectoryPath *string
	PageSize      *int64
	PageToken     *string
}

List directory contents.

type ListDirectoryResponse 

type ListDirectoryResponse struct {
	Contents      []DirectoryEntry
	NextPageToken *string
}

type ListStatusRequest 

type ListStatusRequest struct {
	Path *string
}

type ListStatusResponse 

type ListStatusResponse struct {
	Files []FileInfo
}

type MkDirsRequest 

type MkDirsRequest struct {
	Path *string
}

type MkDirsResponse 

type MkDirsResponse struct {
}

type MoveRequest 

type MoveRequest struct {
	SourcePath      *string
	DestinationPath *string
}

type MoveResponse 

type MoveResponse struct {
}

type Progress 

type Progress struct {
	// Transferred is the cumulative number of bytes confirmed uploaded so far.
	Transferred int64
	// Total is the total size in bytes, or -1 if it is not known in advance (a
	// non-seekable stream).
	Total int64
}

Progress reports the state of an in-flight upload. Fields may be added in future releases, so callers must not depend on the struct being comparable or on its exact size; always refer to fields by name.

type ProgressFunc 

type ProgressFunc func(Progress)

ProgressFunc is invoked as an upload makes progress. It is called from internal goroutines but never concurrently with itself, so it needs no locking of its own; it must return promptly.

type PutRequest 

type PutRequest struct {
	Path      *string
	Contents  []byte
	Overwrite *bool
}

type PutResponse 

type PutResponse struct {
}

type ReadRequest 

type ReadRequest struct {
	Path   *string
	Offset *int64
	Length *int64
}

type ReadResponse 

type ReadResponse struct {
	BytesRead *int64
	Data      []byte
}

type UploadFileRequest 

type UploadFileRequest struct {
	FilePath  *string
	Contents  io.ReadCloser
	Overwrite *bool
}

Upload a file.

type UploadFileResponse 

type UploadFileResponse struct {
}

type UploadOption 

type UploadOption func(*uploadConfig)

UploadOption configures an Upload or UploadFrom call.

func WithLimiter 

func WithLimiter(l Limiter) UploadOption

WithLimiter bounds concurrent transfers via l, shared across Upload calls that pass the same Limiter. When unset, an upload's own parallelism governs its concurrency and there is no cross-upload bound.

func WithOverwrite 

func WithOverwrite(overwrite bool) UploadOption

WithOverwrite controls whether an existing file is overwritten. When this option is not supplied the parameter is omitted and the server applies its default.

func WithParallelism 

func WithParallelism(n int) UploadOption

WithParallelism sets the number of concurrent upload workers used for a large file. A value of 1 uploads the parts sequentially on a single goroutine; higher values upload that many parts at once. It must be at least 1. When not set, a default is used.

func WithPartSize 

func WithPartSize(partSize int64) UploadOption

WithPartSize sets the multipart part size in bytes. It must not exceed the cloud provider maximum. When not supplied an appropriate size is chosen from the content length.

func WithProgress 

func WithProgress(fn ProgressFunc) UploadOption

WithProgress registers a callback invoked as the upload progresses, reporting the cumulative bytes uploaded and the total size (-1 if unknown). It is useful for rendering an upload progress bar.

func WithTransferClient 

func WithTransferClient(client *http.Client) UploadOption

WithTransferClient overrides the HTTP client used to transfer file contents during a large-file upload, replacing the one the call would otherwise create internally. Use it to route the transfer through a custom transport, proxy, or CA.

Because these transfers use self-authenticating presigned URLs, the client must not attach Databricks credentials; the storage provider rejects extra authentication. Do not set http.Client.Timeout, a whole-request deadline that would abort a legitimately long part transfer; bound the upload with the context passed to Upload instead.

type UploadResult 

type UploadResult struct{}

UploadResult holds the result of an upload. It is currently empty and exists for forward compatibility.

Source Files

View all Source files

Directories

Path Synopsis
internal
cloudstorage
Package cloudstorage issues unauthenticated, idempotent HTTP requests to cloud object storage (S3, Azure Blob, GCS) using short-lived presigned URLs.
 Package cloudstorage issues unauthenticated, idempotent HTTP requests to cloud object storage (S3, Azure Blob, GCS) using short-lived presigned URLs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.