files

package
v0.0.1-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (

	// ErrAlreadyExists is returned when an upload targets an existing path with
	// overwrite disabled. It is surfaced as a sentinel so callers detect
	// "already exists" with errors.Is regardless of which protocol produced it.
	ErrAlreadyExists = errors.New("the file being created already exists")
)

Sentinel errors surfaced by the large-file upload API.

Functions

This section is empty.

Types

type AddBlockRequest 

type AddBlockRequest struct {
	// The handle on an open stream.
	Handle *int64
	// The base64-encoded data to append to the stream. This has a limit of 1 MB.
	Data []byte
}

type AddBlockResponse 

type AddBlockResponse struct {
}

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewClient 

func NewClient(ctx context.Context, opts ...client.Option) (*Client, error)

func (*Client) AddBlock 

func (c *Client) AddBlock(ctx context.Context, req *AddBlockRequest, opts ...call.Option) (*AddBlockResponse, error)

Appends a block of data to the stream specified by the input handle. If the handle does not exist, this call will throw an exception with “RESOURCE_DOES_NOT_EXIST“.

If the block of data exceeds 1 MB, this call will throw an exception with “MAX_BLOCK_SIZE_EXCEEDED“.

func (*Client) Close 

func (c *Client) Close(ctx context.Context, req *CloseRequest, opts ...call.Option) (*CloseResponse, error)

Closes the stream specified by the input handle. If the handle does not exist, this call throws an exception with “RESOURCE_DOES_NOT_EXIST“.

func (*Client) Create 

func (c *Client) Create(ctx context.Context, req *CreateRequest, opts ...call.Option) (*CreateResponse, error)

Opens a stream to write to a file and returns a handle to this stream. There is a 10 minute idle timeout on this handle. If a file or directory already exists on the given path and __overwrite__ is set to false, this call will throw an exception with “RESOURCE_ALREADY_EXISTS“.

A typical workflow for file upload would be:

1. Issue a “create“ call and get a handle. 2. Issue one or more “add-block“ calls with the handle you have. 3. Issue a “close“ call with the handle you have.

func (*Client) CreateDirectory 

func (c *Client) CreateDirectory(ctx context.Context, req *CreateDirectoryRequest, opts ...call.Option) (*CreateDirectoryResponse, error)

Creates an empty directory. If necessary, also creates any parent directories of the new, empty directory (like the shell command `mkdir -p`). If called on an existing directory, returns a success response; this method is idempotent (it will succeed if the directory already exists).

func (*Client) Delete 

func (c *Client) Delete(ctx context.Context, req *DeleteRequest, opts ...call.Option) (*DeleteResponse, error)

Delete the file or directory (optionally recursively delete all files in the directory). This call throws an exception with `IO_ERROR` if the path is a non-empty directory and `recursive` is set to `false` or on other similar errors.

When you delete a large number of files, the delete operation is done in increments. The call returns a response after approximately 45 seconds with an error message (503 Service Unavailable) asking you to re-invoke the delete operation until the directory structure is fully deleted.

For operations that delete more than 10K files, we discourage using the DBFS REST API, but advise you to perform such operations in the context of a cluster, using the [File system utility (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs). `dbutils.fs` covers the functional scope of the DBFS REST API, but from notebooks. Running such operations using notebooks provides better control and manageability, such as selective deletes, and the possibility to automate periodic delete jobs.

func (*Client) DeleteDirectory 

func (c *Client) DeleteDirectory(ctx context.Context, req *DeleteDirectoryRequest, opts ...call.Option) (*DeleteDirectoryResponse, error)

Deletes an empty directory.

To delete a non-empty directory, first delete all of its contents. This can be done by listing the directory contents and deleting each file and subdirectory recursively.

func (*Client) DeleteFile 

func (c *Client) DeleteFile(ctx context.Context, req *DeleteFileRequest, opts ...call.Option) (*DeleteFileResponse, error)

Deletes a file. If the request is successful, there is no response body.

func (*Client) DownloadFile 

func (c *Client) DownloadFile(ctx context.Context, req *DownloadFileRequest, opts ...call.Option) (*DownloadFileResponse, error)

Downloads a file. The file contents are the response body. This is a standard HTTP file download, not a JSON RPC. It supports the Range and If-Unmodified-Since HTTP headers.

func (*Client) GetDirectoryMetadata 

func (c *Client) GetDirectoryMetadata(ctx context.Context, req *GetDirectoryMetadataRequest, opts ...call.Option) (*GetDirectoryMetadataResponse, error)

Get the metadata of a directory. The response HTTP headers contain the metadata. There is no response body.

This method is useful to check if a directory exists and the caller has access to it.

If you wish to ensure the directory exists, you can instead use `PUT`, which will create the directory if it does not exist, and is idempotent (it will succeed if the directory already exists).

func (*Client) GetFileMetadata 

func (c *Client) GetFileMetadata(ctx context.Context, req *GetFileMetadataRequest, opts ...call.Option) (*GetFileMetadataResponse, error)

Get the metadata of a file. The response HTTP headers contain the metadata. There is no response body.

func (*Client) GetStatus 

func (c *Client) GetStatus(ctx context.Context, req *GetStatusRequest, opts ...call.Option) (*GetStatusResponse, error)

Gets the file information for a file or directory. If the file or directory does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`.

func (*Client) List 

func (c *Client) List(ctx context.Context, req *ListStatusRequest, opts ...call.Option) (*ListStatusResponse, error)

List the contents of a directory, or details of the file. If the file or directory does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`.

When calling list on a large directory, the list operation will time out after approximately 60 seconds. We strongly recommend using list only on directories containing less than 10K files and discourage using the DBFS REST API for operations that list more than 10K files. Instead, we recommend that you perform such operations in the context of a cluster, using the [File system utility (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs), which provides the same functionality without timing out.

func (*Client) ListDirectoryContents 

func (c *Client) ListDirectoryContents(ctx context.Context, req *ListDirectoryContentsRequest, opts ...call.Option) (*ListDirectoryResponse, error)

Returns the contents of a directory. If there is no directory at the specified path, the API returns an HTTP 404 error.

func (*Client) ListDirectoryContentsIter 

func (c *Client) ListDirectoryContentsIter(ctx context.Context, req *ListDirectoryContentsRequest, opts ...call.Option) iter.Seq2[*DirectoryEntry, error]

ListDirectoryContentsIter returns an iterator that iterates over the results of ListDirectoryContents.

For example: 

for item, err := range c.ListDirectoryContentsIter(ctx, &ListDirectoryContentsRequest{}) {
  if err != nil {
    return err
  }
  fmt.Println(item)
}

Options opts are passed to each ListDirectoryContents call made by the iterator under the hood.

Callers who need custom pagination logic should use ListDirectoryContents directly.

func (*Client) Mkdirs 

func (c *Client) Mkdirs(ctx context.Context, req *MkDirsRequest, opts ...call.Option) (*MkDirsResponse, error)

Creates the given directory and necessary parent directories if they do not exist. If a file (not a directory) exists at any prefix of the input path, this call throws an exception with `RESOURCE_ALREADY_EXISTS`. **Note**: If this operation fails, it might have succeeded in creating some of the necessary parent directories.

func (*Client) Move 

func (c *Client) Move(ctx context.Context, req *MoveRequest, opts ...call.Option) (*MoveResponse, error)

Moves a file from one location to another location within DBFS. If the source file does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. If a file already exists in the destination path, this call throws an exception with `RESOURCE_ALREADY_EXISTS`. If the given source path is a directory, this call always recursively moves all files.

func (*Client) Put 

func (c *Client) Put(ctx context.Context, req *PutRequest, opts ...call.Option) (*PutResponse, error)

Uploads a file through the use of multipart form post. It is mainly used for streaming uploads, but can also be used as a convenient single call for data upload.

Alternatively you can pass contents as base64 string.

The amount of data that can be passed (when not streaming) using the __contents__ parameter is limited to 1 MB. `MAX_BLOCK_SIZE_EXCEEDED` will be thrown if this limit is exceeded.

If you want to upload large files, use the streaming upload. For details, see :method:dbfs/create, :method:dbfs/addBlock, :method:dbfs/close.

func (*Client) Read 

func (c *Client) Read(ctx context.Context, req *ReadRequest, opts ...call.Option) (*ReadResponse, error)

Returns the contents of a file. If the file does not exist, this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. If the path is a directory, the read length is negative, or if the offset is negative, this call throws an exception with `INVALID_PARAMETER_VALUE`. If the read length exceeds 1 MB, this call throws an exception with `MAX_READ_SIZE_EXCEEDED`.

If `offset + length` exceeds the number of bytes in a file, it reads the contents until the end of file.

func (*Client) Upload 

func (c *Client) Upload(ctx context.Context, filePath string, contents io.Reader, opts ...UploadOption) (*UploadResult, error)

Upload uploads contents to the remote file at filePath, which must be an absolute path such as "/Volumes/catalog/schema/volume/file". It chooses a single-shot, multipart, or resumable upload based on the stream size (when the reader is seekable) and the protocol the workspace's Files API selects.

When contents implements io.ReaderAt (for example an *os.File or a *bytes.Reader), parts are read from it with concurrent positioned reads rather than buffered in memory. A reader without random access has each in-flight part buffered as it is read.

ctx bounds the entire operation, including all retries; pass a context with a deadline to cap the total upload time.

func (*Client) UploadFile 

func (c *Client) UploadFile(ctx context.Context, req *UploadFileRequest, opts ...call.Option) (*UploadFileResponse, error)

Uploads a file of up to 5 GiB. The file contents should be sent as the request body as raw bytes (an octet stream); do not encode or otherwise modify the bytes before sending. The contents of the resulting file will be exactly the bytes sent in the request body. If the request is successful, there is no response body.

func (*Client) UploadFrom 

func (c *Client) UploadFrom(ctx context.Context, filePath, sourcePath string, opts ...UploadOption) (*UploadResult, error)

UploadFrom uploads the local file at sourcePath to the remote file at filePath. The local file is opened as needed; its size is always known, so it never takes the non-seekable path.

Parts are read from the file on demand rather than buffered, so the upload covers the file as of its size when the call begins: bytes appended afterward are not included, and truncating the file mid-upload fails the upload (it does not produce a partial object). Do not modify the file while an upload is in progress.

ctx bounds the entire operation, including all retries; pass a context with a deadline to cap the total upload time.

type CloseRequest 

type CloseRequest struct {
	// The handle on an open stream.
	Handle *int64
}

type CloseResponse 

type CloseResponse struct {
}

type CreateDirectoryRequest 

type CreateDirectoryRequest struct {
	// The absolute path of a directory.
	DirectoryPath *string
}

Create a directory.

type CreateDirectoryResponse 

type CreateDirectoryResponse struct {
}

type CreateRequest 

type CreateRequest struct {
	// The path of the new file. The path should be the absolute DBFS path.
	Path *string
	// The flag that specifies whether to overwrite existing file/files.
	Overwrite *bool
}

type CreateResponse 

type CreateResponse struct {
	// Handle which should subsequently be passed into the AddBlock and Close calls
	// when writing to a file through a stream.
	Handle *int64
}

type DeleteDirectoryRequest 

type DeleteDirectoryRequest struct {
	// The absolute path of a directory.
	DirectoryPath *string
}

Delete a directory.

type DeleteDirectoryResponse 

type DeleteDirectoryResponse struct {
}

type DeleteFileRequest 

type DeleteFileRequest struct {
	// The absolute path of the file.
	FilePath *string
}

Delete a file.

type DeleteFileResponse 

type DeleteFileResponse struct {
}

type DeleteRequest 

type DeleteRequest struct {
	// The path of the file or directory to delete. The path should be the absolute
	// DBFS path.
	Path *string
	// Whether or not to recursively delete the directory's contents. Deleting empty
	// directories can be done without providing the recursive flag.
	Recursive *bool
}

type DeleteResponse 

type DeleteResponse struct {
}

type DirectoryEntry 

type DirectoryEntry struct {
	// The length of the file in bytes. This field is omitted for directories.
	FileSize *int
	// True if the path is a directory.
	IsDirectory *bool
	// Last modification time of given file in milliseconds since unix epoch.
	LastModified *int
	// The name of the file or directory. This is the last component of the path.
	Name *string
	// The absolute path of the file or directory.
	Path *string
}

type DownloadFileRequest 

type DownloadFileRequest struct {
	// The absolute path of the file.
	FilePath *string
	// The range of bytes to retrieve. The range is inclusive and zero-based, see
	// [RFC 9110] for further details.
	//
	// [RFC 9110]: https://datatracker.ietf.org/doc/html/rfc9110#name-range
	Range *string
	// Download the file only if it has not been modified since the specified
	// timestamp. If it has, a 412 Precondition Failed error will be returned. See
	// [RFC 9110] for further details.
	//
	// [RFC 9110]: https://datatracker.ietf.org/doc/html/rfc9110#name-if-unmodified-since
	IfUnmodifiedSince *string
}

Download a file.

type DownloadFileResponse 

type DownloadFileResponse struct {
	// The length of the HTTP response body in bytes.
	ContentLength *int64
	ContentType   *string
	Contents      io.ReadCloser
	// The last modified time of the file in HTTP-date (RFC 7231) format.
	LastModified *string
}

type FileInfo 

type FileInfo struct {
	// The absolute path of the file or directory.
	Path *string
	// True if the path is a directory.
	IsDir *bool
	// The length of the file in bytes. Set to 0 for directories.
	FileSize *int64
	// Last modification time of given file in milliseconds since epoch.
	ModificationTime *int64
}

Stores the attributes of a file or directory..

type GetDirectoryMetadataRequest 

type GetDirectoryMetadataRequest struct {
	// The absolute path of a directory.
	DirectoryPath *string
}

Get directory metadata.

type GetDirectoryMetadataResponse 

type GetDirectoryMetadataResponse struct {
}

type GetFileMetadataRequest 

type GetFileMetadataRequest struct {
	// The absolute path of the file.
	FilePath *string
	// The range of bytes to retrieve. The range is inclusive and zero-based, see
	// [RFC 9110] for further details.
	//
	// [RFC 9110]: https://datatracker.ietf.org/doc/html/rfc9110#name-range
	Range *string
	// Download the file only if it has not been modified since the specified
	// timestamp. If it has, a 412 Precondition Failed error will be returned. See
	// [RFC 9110] for further details.
	//
	// [RFC 9110]: https://datatracker.ietf.org/doc/html/rfc9110#name-if-unmodified-since
	IfUnmodifiedSince *string
}

Get file metadata.

type GetFileMetadataResponse 

type GetFileMetadataResponse struct {
	// The length of the HTTP response body in bytes.
	ContentLength *int64
	ContentType   *string
	// The last modified time of the file in HTTP-date (RFC 7231) format.
	LastModified *string
}

type GetStatusRequest 

type GetStatusRequest struct {
	// The path of the file or directory. The path should be the absolute DBFS path.
	Path *string
}

type GetStatusResponse 

type GetStatusResponse struct {
	// The absolute path of the file or directory.
	Path *string
	// True if the path is a directory.
	IsDir *bool
	// The length of the file in bytes. Set to 0 for directories.
	FileSize *int64
	// Last modification time of given file in milliseconds since epoch.
	ModificationTime *int64
}

type Limiter 

type Limiter interface {
	// Acquire blocks until a slot is free or ctx is cancelled; on cancellation it
	// returns ctx.Err() and the caller must not Release.
	Acquire(ctx context.Context) error
	// Release returns a slot taken by a successful Acquire.
	Release()
}

Limiter bounds the number of concurrent cloud-leg transfers an upload may run. The engine acquires one unit before each transfer (every multipart part PUT, the single-shot PUT, and every resumable chunk) and releases it when that transfer returns. Pass the same Limiter to multiple Upload calls -- for example when copying many files at once -- to cap their combined concurrency. Implementations must be safe for concurrent use.

func NewLimiter 

func NewLimiter(n int) Limiter

NewLimiter returns a Limiter permitting at most n concurrent transfers. A value of n <= 0 yields an unlimited limiter whose Acquire never blocks.

type ListDirectoryContentsRequest 

type ListDirectoryContentsRequest struct {
	// The absolute path of a directory.
	DirectoryPath *string
	// The maximum number of directory entries to return. The response may contain
	// fewer entries. If the response contains a `next_page_token`, there may be
	// more entries, even if fewer than `page_size` entries are in the response.
	//
	// We recommend not to set this value unless you are intentionally listing less
	// than the complete directory contents.
	//
	// If unspecified, at most 1000 directory entries will be returned. The maximum
	// value is 1000. Values above 1000 will be coerced to 1000.
	PageSize *int64
	// An opaque page token which was the `next_page_token` in the response of the
	// previous request to list the contents of this directory. Provide this token
	// to retrieve the next page of directory entries. When providing a
	// `page_token`, all other parameters provided to the request must match the
	// previous request. To list all of the entries in a directory, it is necessary
	// to continue requesting pages of entries until the response contains no
	// `next_page_token`. Note that the number of entries returned must not be used
	// to determine when the listing is complete.
	PageToken *string
}

List directory contents.

type ListDirectoryResponse 

type ListDirectoryResponse struct {
	// Array of DirectoryEntry.
	Contents []DirectoryEntry
	// A token, which can be sent as `page_token` to retrieve the next page.
	NextPageToken *string
}

type ListStatusRequest 

type ListStatusRequest struct {
	// The path of the file or directory. The path should be the absolute DBFS path.
	Path *string
}

type ListStatusResponse 

type ListStatusResponse struct {
	// A list of FileInfo's that describe contents of directory or file. See example
	// above.
	Files []FileInfo
}

type MkDirsRequest 

type MkDirsRequest struct {
	// The path of the new directory. The path should be the absolute DBFS path.
	Path *string
}

type MkDirsResponse 

type MkDirsResponse struct {
}

type MoveRequest 

type MoveRequest struct {
	// The source path of the file or directory. The path should be the absolute
	// DBFS path.
	SourcePath *string
	// The destination path of the file or directory. The path should be the
	// absolute DBFS path.
	DestinationPath *string
}

type MoveResponse 

type MoveResponse struct {
}

type Progress 

type Progress struct {
	// Transferred is the cumulative number of bytes confirmed uploaded so far.
	Transferred int64
	// Total is the total size in bytes, or -1 if it is not known in advance (a
	// non-seekable stream).
	Total int64
}

Progress reports the state of an in-flight upload. Fields may be added in future releases, so callers must not depend on the struct being comparable or on its exact size; always refer to fields by name.

type ProgressFunc 

type ProgressFunc func(Progress)

ProgressFunc is invoked as an upload makes progress. It is called from internal goroutines but never concurrently with itself, so it needs no locking of its own; it must return promptly.

type PutRequest 

type PutRequest struct {
	// The path of the new file. The path should be the absolute DBFS path.
	Path *string
	// This parameter might be absent, and instead a posted file will be used.
	Contents []byte
	// The flag that specifies whether to overwrite existing file/files.
	Overwrite *bool
}

type PutResponse 

type PutResponse struct {
}

type ReadRequest 

type ReadRequest struct {
	// The path of the file to read. The path should be the absolute DBFS path.
	Path *string
	// The offset to read from in bytes.
	Offset *int64
	// The number of bytes to read starting from the offset. This has a limit of 1
	// MB, and a default value of 0.5 MB.
	Length *int64
}

type ReadResponse 

type ReadResponse struct {
	// The number of bytes read (could be less than “length“ if we hit end of
	// file). This refers to number of bytes read in unencoded version (response
	// data is base64-encoded).
	BytesRead *int64
	// The base64-encoded contents of the file read.
	Data []byte
}

type UploadFileRequest 

type UploadFileRequest struct {
	// The absolute path of the file.
	FilePath *string
	Contents io.ReadCloser
	// If true or unspecified, an existing file will be overwritten. If false, an
	// error will be returned if the path points to an existing file.
	Overwrite *bool
}

Upload a file.

type UploadFileResponse 

type UploadFileResponse struct {
}

type UploadOption 

type UploadOption func(*uploadConfig)

UploadOption configures an Upload or UploadFrom call.

func WithLimiter 

func WithLimiter(l Limiter) UploadOption

WithLimiter bounds concurrent transfers via l, shared across Upload calls that pass the same Limiter. When unset, an upload's own parallelism governs its concurrency and there is no cross-upload bound.

func WithOverwrite 

func WithOverwrite(overwrite bool) UploadOption

WithOverwrite controls whether an existing file is overwritten. When this option is not supplied the parameter is omitted and the server applies its default.

func WithParallelism 

func WithParallelism(n int) UploadOption

WithParallelism sets the number of concurrent upload workers used for a large file. A value of 1 uploads the parts sequentially on a single goroutine; higher values upload that many parts at once. It must be at least 1. When not set, a default is used.

func WithPartSize 

func WithPartSize(partSize int64) UploadOption

WithPartSize sets the multipart part size in bytes. It must not exceed the cloud provider maximum. When not supplied an appropriate size is chosen from the content length.

func WithProgress 

func WithProgress(fn ProgressFunc) UploadOption

WithProgress registers a callback invoked as the upload progresses, reporting the cumulative bytes uploaded and the total size (-1 if unknown). It is useful for rendering an upload progress bar.

func WithTransferClient 

func WithTransferClient(client *http.Client) UploadOption

WithTransferClient overrides the HTTP client used to transfer file contents during a large-file upload, replacing the one the call would otherwise create internally. Use it to route the transfer through a custom transport, proxy, or CA.

Because these transfers use self-authenticating presigned URLs, the client must not attach Databricks credentials; the storage provider rejects extra authentication. Do not set http.Client.Timeout, a whole-request deadline that would abort a legitimately long part transfer; bound the upload with the context passed to Upload instead.

type UploadResult 

type UploadResult struct{}

UploadResult holds the result of an upload. It is currently empty and exists for forward compatibility.

Source Files

View all Source files

Directories

Path Synopsis
internal
cloudstorage
Package cloudstorage issues unauthenticated, idempotent HTTP requests to cloud object storage (S3, Azure Blob, GCS) using short-lived presigned URLs.
 Package cloudstorage issues unauthenticated, idempotent HTTP requests to cloud object storage (S3, Azure Blob, GCS) using short-lived presigned URLs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.