dotc1z

package
v0.6.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 14, 2026 License: Apache-2.0 Imports: 35 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidFile        = fmt.Errorf("c1z: invalid file")
	ErrMaxSizeExceeded    = fmt.Errorf("c1z: max decoded size exceeded, increase DecoderMaxDecodedSize using %v environment variable", maxDecodedSizeEnvVar)
	ErrWindowSizeExceeded = fmt.Errorf("c1z: window size exceeded, increase DecoderMaxMemory using %v  environment variable", maxDecoderMemorySizeEnv)
)
View Source 
var C1ZFileHeader = []byte("C1ZF\x00")
View Source 
var ErrReadOnly = errors.New("c1z: read only mode")

Functions

func C1ZFileCheckHeader 

func C1ZFileCheckHeader(f io.ReadSeeker) (bool, error)

C1ZFileCheckHeader reads len(C1ZFileHeader) bytes from the given io.ReadSeeker and compares them to C1ZFileHeader. Returns true if the header is valid. Returns any errors from Read() or Seek(). If a nil error is returned, the given io.ReadSeeker will be pointing to the first byte of the stream, and is suitable to be passed to NewC1ZFileDecoder.

func NewC1FileReader 

func NewC1FileReader(ctx context.Context, dbFilePath string, opts ...C1FOption) (connectorstore.Reader, error)

NewC1FileReader returns a connectorstore.Reader implementation for the given sqlite db file path.

func NewC1ZFileDecoder 

func NewC1ZFileDecoder(f io.Reader, opts ...DecoderOption) (io.ReadCloser, error)

NewC1ZFileDecoder wraps a given .c1z io.Reader that validates the .c1z and decompresses/decodes the underlying file. Defaults: 128MiB max memory and 3GiB max decoded size You must close the resulting io.ReadCloser when you are done, do not forget to close the given io.Reader if necessary.

func NewDecoder added in v0.0.24 

func NewDecoder(f io.Reader, opts ...DecoderOption) (*decoder, error)

NewDecoder wraps a given .c1z file io.Reader and returns an io.Reader for the underlying decoded/uncompressed file.

func NewExternalC1FileReader added in v0.2.84 

func NewExternalC1FileReader(ctx context.Context, tmpDir string, externalResourceC1ZPath string) (connectorstore.Reader, error)

func ReadHeader added in v0.0.24 

func ReadHeader(reader io.Reader) error

ReadHeader reads len(C1ZFileHeader) bytes from the given io.Reader and compares them to C1ZFileHeader, returning an error if they don't match. If possible, ReadHeader will Seek() to the start of the stream before checking the header bytes. On return, the reader will be pointing to the first byte after the header.

Types

type C1FOption added in v0.1.8 

type C1FOption func(*C1File)

func WithC1FEncoderConcurrency added in v0.6.6 

func WithC1FEncoderConcurrency(concurrency int) C1FOption

func WithC1FPragma added in v0.1.22 

func WithC1FPragma(name string, value string) C1FOption

WithC1FPragma sets a sqlite pragma for the c1z file.

func WithC1FReadOnly added in v0.6.6 

func WithC1FReadOnly(readOnly bool) C1FOption

func WithC1FTmpDir added in v0.1.8 

func WithC1FTmpDir(tempDir string) C1FOption

WithC1FTmpDir sets the temporary directory to use when cloning a sync. If not provided, os.TempDir() will be used.

type C1File added in v0.0.24 

type C1File struct {
	// contains filtered or unexported fields
}

func NewC1File added in v0.0.24 

func NewC1File(ctx context.Context, dbFilePath string, opts ...C1FOption) (*C1File, error)

Returns a C1File instance for the given db filepath.

func NewC1ZFile added in v0.0.24 

func NewC1ZFile(ctx context.Context, outputFilePath string, opts ...C1ZOption) (*C1File, error)

Returns a new C1File instance with its state stored at the provided filename.

func (*C1File) AttachFile added in v0.3.35 

func (c *C1File) AttachFile(other *C1File, dbName string) (*C1FileAttached, error)

func (*C1File) CheckpointSync added in v0.0.24 

func (c *C1File) CheckpointSync(ctx context.Context, syncToken string) error

func (*C1File) Cleanup added in v0.0.24 

func (c *C1File) Cleanup(ctx context.Context) error

func (*C1File) Clear added in v0.5.0 

func (c *C1File) Clear(ctx context.Context, opt ...sessions.SessionStoreOption) error

Clear implements types.SessionStore.

func (*C1File) CloneSync added in v0.0.24 

func (c *C1File) CloneSync(ctx context.Context, outPath string, syncID string) (err error)

CloneSync uses sqlite hackery to directly copy the pertinent rows into a new database. 1. Create a new empty sqlite database in a temp file 2. Open the c1z that we are cloning to get a db handle 3. Execute an ATTACH query to bring our empty sqlite db into the context of our db connection 4. Select directly from the cloned db and insert directly into the new database. 5. Close and save the new database as a c1z at the configured path.

func (*C1File) Close added in v0.0.24 

func (c *C1File) Close(ctx context.Context) error

Close ensures that the sqlite database is flushed to disk, and if any changes were made we update the original database with our changes. The provided context is used for the WAL checkpoint operation.

func (*C1File) CurrentSyncStep added in v0.0.24 

func (c *C1File) CurrentSyncStep(ctx context.Context) (string, error)

func (*C1File) Delete added in v0.5.0 

func (c *C1File) Delete(ctx context.Context, key string, opt ...sessions.SessionStoreOption) error

Delete implements types.SessionStore.

func (*C1File) DeleteGrant added in v0.2.84 

func (c *C1File) DeleteGrant(ctx context.Context, grantId string) error

func (*C1File) DeleteSyncRun added in v0.0.24 

func (c *C1File) DeleteSyncRun(ctx context.Context, syncID string) error

DeleteSyncRun removes all the objects with a given syncID from the database.

func (*C1File) EndSync added in v0.0.24 

func (c *C1File) EndSync(ctx context.Context) error

EndSync updates the current sync_run row with the end time, and removes any other objects that don't have the current sync ID.

func (*C1File) GenerateSyncDiff added in v0.3.2 

func (c *C1File) GenerateSyncDiff(ctx context.Context, baseSyncID string, appliedSyncID string) (string, error)

func (*C1File) Get added in v0.5.0 

func (c *C1File) Get(ctx context.Context, key string, opt ...sessions.SessionStoreOption) ([]byte, bool, error)

Get implements types.SessionCache.

func (*C1File) GetAll added in v0.5.0 

func (c *C1File) GetAll(ctx context.Context, pageToken string, opt ...sessions.SessionStoreOption) (map[string][]byte, string, error)

GetAll implements types.SessionStore.

func (*C1File) GetAsset added in v0.0.24 

func (c *C1File) GetAsset(ctx context.Context, request *v2.AssetServiceGetAssetRequest) (string, io.Reader, error)

GetAsset fetches the specified asset from the database, and returns the content type and an io.Reader for the caller to read the asset from.

func (*C1File) GetEntitlement added in v0.0.24 

func (c *C1File) GetEntitlement(ctx context.Context, request *reader_v2.EntitlementsReaderServiceGetEntitlementRequest) (*reader_v2.EntitlementsReaderServiceGetEntitlementResponse, error)

func (*C1File) GetGrant added in v0.0.24 

func (c *C1File) GetGrant(ctx context.Context, request *reader_v2.GrantsReaderServiceGetGrantRequest) (*reader_v2.GrantsReaderServiceGetGrantResponse, error)

func (*C1File) GetLatestFinishedSync added in v0.2.83 

func (c *C1File) GetLatestFinishedSync(ctx context.Context, request *reader_v2.SyncsReaderServiceGetLatestFinishedSyncRequest) (*reader_v2.SyncsReaderServiceGetLatestFinishedSyncResponse, error)

func (*C1File) GetMany added in v0.5.0 

func (c *C1File) GetMany(ctx context.Context, keys []string, opt ...sessions.SessionStoreOption) (map[string][]byte, []string, error)

GetMany implements types.SessionStore.

func (*C1File) GetResource added in v0.0.24 

func (c *C1File) GetResource(ctx context.Context, request *reader_v2.ResourcesReaderServiceGetResourceRequest) (*reader_v2.ResourcesReaderServiceGetResourceResponse, error)

func (*C1File) GetResourceType added in v0.0.24 

func (c *C1File) GetResourceType(ctx context.Context, request *reader_v2.ResourceTypesReaderServiceGetResourceTypeRequest) (*reader_v2.ResourceTypesReaderServiceGetResourceTypeResponse, error)

func (*C1File) GetSync added in v0.2.83 

func (c *C1File) GetSync(ctx context.Context, request *reader_v2.SyncsReaderServiceGetSyncRequest) (*reader_v2.SyncsReaderServiceGetSyncResponse, error)

func (*C1File) GrantStats added in v0.3.58 

func (c *C1File) GrantStats(ctx context.Context, syncType connectorstore.SyncType, syncId string) (map[string]int64, error)

GrantStats introspects the database and returns the count of grants for the given sync run. If syncId is empty, it will use the latest sync run of the given type.

func (*C1File) InitTables added in v0.6.5 

func (c *C1File) InitTables(ctx context.Context) error

func (*C1File) LatestFinishedSyncID added in v0.3.56 

func (c *C1File) LatestFinishedSyncID(ctx context.Context, syncType connectorstore.SyncType) (string, error)

func (*C1File) LatestSyncID added in v0.0.24 

func (c *C1File) LatestSyncID(ctx context.Context, syncType connectorstore.SyncType) (string, error)

func (*C1File) ListEntitlements added in v0.0.24 

func (c *C1File) ListEntitlements(ctx context.Context, request *v2.EntitlementsServiceListEntitlementsRequest) (*v2.EntitlementsServiceListEntitlementsResponse, error)

func (*C1File) ListGrants added in v0.0.24 

func (c *C1File) ListGrants(ctx context.Context, request *v2.GrantsServiceListGrantsRequest) (*v2.GrantsServiceListGrantsResponse, error)

func (*C1File) ListGrantsForEntitlement added in v0.0.24 

func (c *C1File) ListGrantsForEntitlement(
	ctx context.Context,
	request *reader_v2.GrantsReaderServiceListGrantsForEntitlementRequest,
) (*reader_v2.GrantsReaderServiceListGrantsForEntitlementResponse, error)

func (*C1File) ListGrantsForPrincipal added in v0.1.6 

func (c *C1File) ListGrantsForPrincipal(
	ctx context.Context,
	request *reader_v2.GrantsReaderServiceListGrantsForEntitlementRequest,
) (*reader_v2.GrantsReaderServiceListGrantsForEntitlementResponse, error)

func (*C1File) ListGrantsForResourceType added in v0.0.24 

func (c *C1File) ListGrantsForResourceType(
	ctx context.Context,
	request *reader_v2.GrantsReaderServiceListGrantsForResourceTypeRequest,
) (*reader_v2.GrantsReaderServiceListGrantsForResourceTypeResponse, error)

func (*C1File) ListResourceTypes added in v0.0.24 

func (c *C1File) ListResourceTypes(ctx context.Context, request *v2.ResourceTypesServiceListResourceTypesRequest) (*v2.ResourceTypesServiceListResourceTypesResponse, error)

func (*C1File) ListResources added in v0.0.24 

func (c *C1File) ListResources(ctx context.Context, request *v2.ResourcesServiceListResourcesRequest) (*v2.ResourcesServiceListResourcesResponse, error)

func (*C1File) ListStaticEntitlements added in v0.5.21 

func (c *C1File) ListStaticEntitlements(ctx context.Context, request *v2.EntitlementsServiceListStaticEntitlementsRequest) (*v2.EntitlementsServiceListStaticEntitlementsResponse, error)

func (*C1File) ListSyncRuns added in v0.0.24 

func (c *C1File) ListSyncRuns(ctx context.Context, pageToken string, pageSize uint32) ([]*syncRun, string, error)

func (*C1File) ListSyncs added in v0.2.83 

func (c *C1File) ListSyncs(ctx context.Context, request *reader_v2.SyncsReaderServiceListSyncsRequest) (*reader_v2.SyncsReaderServiceListSyncsResponse, error)

func (*C1File) OutputFilepath added in v0.3.3 

func (c *C1File) OutputFilepath() (string, error)

func (*C1File) PreviousSyncID added in v0.0.24 

func (c *C1File) PreviousSyncID(ctx context.Context, syncType connectorstore.SyncType) (string, error)

func (*C1File) PutAsset added in v0.0.24 

func (c *C1File) PutAsset(ctx context.Context, assetRef *v2.AssetRef, contentType string, data []byte) error

PutAsset stores the given asset in the database.

func (*C1File) PutEntitlements added in v0.1.46 

func (c *C1File) PutEntitlements(ctx context.Context, entitlementObjs ...*v2.Entitlement) error

func (*C1File) PutEntitlementsIfNewer added in v0.3.3 

func (c *C1File) PutEntitlementsIfNewer(ctx context.Context, entitlementObjs ...*v2.Entitlement) error

func (*C1File) PutGrants added in v0.1.46 

func (c *C1File) PutGrants(ctx context.Context, bulkGrants ...*v2.Grant) error

func (*C1File) PutGrantsIfNewer added in v0.3.3 

func (c *C1File) PutGrantsIfNewer(ctx context.Context, bulkGrants ...*v2.Grant) error

func (*C1File) PutResourceTypes added in v0.1.46 

func (c *C1File) PutResourceTypes(ctx context.Context, resourceTypesObjs ...*v2.ResourceType) error

func (*C1File) PutResourceTypesIfNewer added in v0.3.3 

func (c *C1File) PutResourceTypesIfNewer(ctx context.Context, resourceTypesObjs ...*v2.ResourceType) error

func (*C1File) PutResources added in v0.1.46 

func (c *C1File) PutResources(ctx context.Context, resourceObjs ...*v2.Resource) error

func (*C1File) PutResourcesIfNewer added in v0.3.3 

func (c *C1File) PutResourcesIfNewer(ctx context.Context, resourceObjs ...*v2.Resource) error

func (*C1File) ResumeSync added in v0.3.56 

func (c *C1File) ResumeSync(ctx context.Context, syncType connectorstore.SyncType, syncID string) (string, error)

func (*C1File) Set added in v0.5.0 

func (c *C1File) Set(ctx context.Context, key string, value []byte, opt ...sessions.SessionStoreOption) error

Set implements types.SessionStore.

func (*C1File) SetCurrentSync added in v0.3.8 

func (c *C1File) SetCurrentSync(ctx context.Context, syncID string) error

func (*C1File) SetMany added in v0.5.0 

func (c *C1File) SetMany(ctx context.Context, values map[string][]byte, opt ...sessions.SessionStoreOption) error

SetMany implements types.SessionStore.

func (*C1File) SetSyncID added in v0.6.8 

func (c *C1File) SetSyncID(_ context.Context, syncID string) error

SetSyncID sets the current sync ID. This is only intended for testing.

func (*C1File) StartNewSync added in v0.2.15 

func (c *C1File) StartNewSync(ctx context.Context, syncType connectorstore.SyncType, parentSyncID string) (string, error)

func (*C1File) StartOrResumeSync added in v0.3.52 

func (c *C1File) StartOrResumeSync(ctx context.Context, syncType connectorstore.SyncType, syncID string) (string, bool, error)

StartOrResumeSync checks if a sync is already running and resumes it if it is. If no sync is running, it starts a new sync. It returns the sync ID and a boolean indicating if a new sync was started.

func (*C1File) Stats added in v0.0.24 

func (c *C1File) Stats(ctx context.Context, syncType connectorstore.SyncType, syncId string) (map[string]int64, error)

Stats introspects the database and returns the count of objects for the given sync run. If syncId is empty, it will use the latest sync run of the given type.

func (*C1File) Vacuum added in v0.1.8 

func (c *C1File) Vacuum(ctx context.Context) error

Vacuum runs a VACUUM on the database to reclaim space.

func (*C1File) ViewSync added in v0.0.24 

func (c *C1File) ViewSync(ctx context.Context, syncID string) error

type C1FileAttached added in v0.3.35 

type C1FileAttached struct {
	// contains filtered or unexported fields
}

func (*C1FileAttached) CompactEntitlements added in v0.3.35 

func (c *C1FileAttached) CompactEntitlements(ctx context.Context, baseSyncID string, appliedSyncID string) error

func (*C1FileAttached) CompactGrants added in v0.3.35 

func (c *C1FileAttached) CompactGrants(ctx context.Context, baseSyncID string, appliedSyncID string) error

func (*C1FileAttached) CompactResourceTypes added in v0.3.35 

func (c *C1FileAttached) CompactResourceTypes(ctx context.Context, baseSyncID string, appliedSyncID string) error

func (*C1FileAttached) CompactResources added in v0.3.35 

func (c *C1FileAttached) CompactResources(ctx context.Context, baseSyncID string, appliedSyncID string) error

func (*C1FileAttached) CompactTable added in v0.3.35 

func (c *C1FileAttached) CompactTable(ctx context.Context, baseSyncID string, appliedSyncID string, tableName string) error

func (*C1FileAttached) DetachFile added in v0.3.35 

func (c *C1FileAttached) DetachFile(dbName string) (*C1FileAttached, error)

func (*C1FileAttached) UpdateSync added in v0.6.11 

func (c *C1FileAttached) UpdateSync(ctx context.Context, baseSync *reader_v2.SyncRun, appliedSync *reader_v2.SyncRun) error

type C1ZOption added in v0.1.8 

type C1ZOption func(*c1zOptions)

func WithDecoderOptions added in v0.3.50 

func WithDecoderOptions(opts ...DecoderOption) C1ZOption

func WithEncoderConcurrency added in v0.6.6 

func WithEncoderConcurrency(concurrency int) C1ZOption

WithEncoderConcurrency sets the number of created encoders. Default is 1, which disables async encoding/concurrency. 0 uses GOMAXPROCS.

func WithPragma added in v0.1.22 

func WithPragma(name string, value string) C1ZOption

WithPragma sets a sqlite pragma for the c1z file.

func WithReadOnly added in v0.6.6 

func WithReadOnly(readOnly bool) C1ZOption

WithReadOnly opens the c1z file in read only mode. Modifying the c1z will result in an error on close.

func WithTmpDir added in v0.1.8 

func WithTmpDir(tmpDir string) C1ZOption

WithTmpDir sets the temporary directory to extract the c1z file to. If not provided, os.TempDir() will be used.

type DecoderOption added in v0.0.24 

type DecoderOption func(*decoderOptions) error

DecoderOption is an option for creating a decoder.

func WithContext 

func WithContext(ctx context.Context) DecoderOption

WithContext sets a context, when cancelled, will cause subequent calls to Read() to return ctx.Error().

func WithDecoderConcurrency added in v0.3.50 

func WithDecoderConcurrency(n int) DecoderOption

WithDecoderConcurrency sets the number of created decoders. Default is 1, which disables async decoding/concurrency. 0 uses GOMAXPROCS. -1 uses GOMAXPROCS or 4, whichever is lower.

func WithDecoderMaxDecodedSize 

func WithDecoderMaxDecodedSize(n uint64) DecoderOption

WithDecoderMaxDecodedSize sets the maximum size of the decoded stream. This can be used to cap the resulting decoded stream size. Maximum is 1 << 63 bytes. Default is 1GiB.

func WithDecoderMaxMemory 

func WithDecoderMaxMemory(n uint64) DecoderOption

WithDecoderMaxMemory sets the maximum window size for streaming operations. This can be used to control memory usage of potentially hostile content. Maximum is 1 << 63 bytes. Default is 128MiB.

type SessionStore added in v0.5.0 

type SessionStore interface {
	sessions.SessionStore
}

Source Files

View all Source files

Directories

Path Synopsis
local
s3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.