README
¶
blobs
An application's files, in blobs/ inside a tinystore.Store: avatars and
their sizes, a chat's attachments, a camera's recordings, an export a user
downloads tomorrow, an editor's documents. Objects sit under keys that are
paths, their rows in blobs.db, their bytes inline up to 16 KiB and in a file
of their own above it. The design and the measurements behind it are
research's design/blobs.md and
the mechanics round.
objects, err := blobs.Open(ctx, store, blobs.Options{}) // data/blobs/
avatars, err := blobs.OpenBucket(ctx, objects, "avatars", blobs.MaxSize(20<<20))
exports, err := blobs.OpenBucket(ctx, objects, "exports", blobs.DefaultTTL(24*time.Hour))
obj, err := avatars.Of(user.ID).Put(ctx, "original", r.Body,
blobs.ContentType(r.Header.Get("Content-Type")), blobs.Size(r.ContentLength))
photo, found, err := avatars.Of(user.ID).Open(ctx, "original") // Read, ReadAt, Seek, Close
http.ServeContent(w, r, "", photo.Modified, photo) // Range, If-Range, If-None-Match, HEAD
upload, err := exports.Of(userID).Create(ctx, "notes.zip", blobs.ContentType("application/zip"))
defer upload.Abort()
_, err = io.Copy(upload, archive)
export, err := upload.Commit(ctx)
_, err = files.Of("users", 7).Move(ctx, "pending/u1", "sent/u1") // one write, the bytes untouched
usage, err := files.Of("users", 7).Usage(ctx) // counted from the rows
err = avatars.Of(user.ID).Clear(ctx) // the account deleted
example_test.go runs the five cases of
design/blobs.md as examples, and go doc shows them.
Contracts
- A key is a path. Segments joined by
/, each non-empty, not.or.., UTF-8 without a control character; at most 16 segments and 1 KiB with the owners. It names no file, soPhoto.jpgandphoto.jpgare two keys on every file system and a name Windows reserves is a key like any other.Of(owners...)names a folder a segment an owner, a string or an integer, an integer by its decimal text;Of("users", 4)never meetsusers/42/. An object'sKeyis its path under the handle that returned it. Anything else istinystore.ErrInvalid. - An object appears at its commit, whole, or not at all.
Putreturns once the object is durable: up to 16 KiB its bytes are a row that commits beside the writes of other goroutines; above it they are a file, written inuploads/, synced, renamed intoobjects/, its directory synced, and only then named by the row. Whatever the key held is replaced at that commit. A file is never written again nor given another name. - Memory does not follow size. An upload holds 16 KiB of the store's
memory while it may yet be inline; past it the bytes go to its file as they
arrive, synced every 256 MiB. After 256 KiB,
Puttries a 64 KiB transfer buffer once, reserving both buffers during the change (80 KiB). If that memory is not free immediately, it keeps the 16 KiB buffer without waiting.Create/Writeneeds only the initial buffer.Size(n)says the stream's length; a shorter or longer stream isErrInvalid. A*bytes.Reader,*bytes.Bufferor*strings.Readersays its own.MaxSize(n)bounds a bucket's objects andOptions.KeepFreewhat an upload leaves free on the disk, 1 GiB unless it says, checked at the start for a declared size and every 64 MiB; either stops the upload withErrLimit. - An upload that does not commit leaves nothing.
Abort, a failedWrite, a stream that fails, the end of the context given toCreateand the store'sCloseeach remove its bytes. A call after its context ended finds it aborted, and maintenance aborts one left without calls. - A condition is checked twice.
IfNoneMatch()creates only;IfMatchtakes anIf-Matchheader as it is, a list of ETags or*, and writes or deletes only while the key holds a live object with one of them. Both are checked when an upload begins, so that a doomed one stops before its first byte, and in the commit, which decides:ErrConflict. - The ETag is the bytes'. It is quoted, as a header carries it, and derives from the SHA-256 computed as the bytes arrive, so equal bytes have one ETag; its derivation is not promised.
- A reader keeps what it opened.
Openreturns the object as it is at that moment: a reader of a file holds the file, opened so that it shares its deletion on Windows, and reads on after its key is deleted, replaced, expired or cleared and after the store closes, until its ownClose. A file gone between the lookup and the open means its key changed: the key is looked up again, three times at most, thenErrConflict; a file missing while its key names it isErrCorrupt. - A whole read is checked, a range is not. An inline object is checked
against its SHA-256 before its first byte. A file read from its first byte
to its last in order, as
io.Copyandhttp.ServeContentread it, fails withErrCorruptinstead of handing over its last bytes when they do not hash to the object's.ReadAt, and aReadafter aSeekelsewhere, read their bytes as the file has them. - The scrub reads every content once a pass. Each
Maintainreads a 43,200th of the contents' bytes and at least 1 MiB, a pass of 30 days at a call a minute, and keeps its place and its hash's state inblobs.db. A content changed or missing is marked, logged once at Error with the keys naming it, and its nextOpenisErrCorrupt; writes over it work as over any other, and a backup is what repairs it. Copyshares the bytes. The destination is a new version naming the source's content, with its type and meta unless the call gives others; no byte is read or written.MoveisCopyand the source's removal in one commit. Both stay inside a bucket, takeIfNoneMatchorIfMatchfor the destination, and refuse an absent source withErrConflict. Bytes leave once no key names them: a file right after the commit that dropped its last name, and at the next maintenance if its removal failed.- Every write gives its object the expiry a
Putwould: the call'sTTLorExpireAt, else the bucket'sDefaultTTLfrom the write, else none. An expired object is absent to every call, and maintenance removes it, 10,000 a transaction. Scanwalks every key under a folder, the folders under it included, in the byte order of the paths,10.jpgbefore9.jpg: 1000 objects a page at most, each page one statement's snapshot;Allwalks the pages holding none between them.Usagecounts the objects and bytes under a folder from their rows, a millisecond over ten thousand; it is a check and not a limit.Clearremoves a folder and every folder under it at once. Up to 10,000 objects go in one transaction; a larger folder is marked cleared at the file's revision, which hides its objects from every call the moment it commits, and maintenance deletes them. An object written after it stays.- A snapshot links the files.
Store.Snapshot, and sobackup.Write, copiesblobs.dband hard-links every file its copy names, a copy where links are refused; removal waits meanwhile. Each file is its own entry of the backup, stored rather than deflated. - An error names its object. A call refused because of one object is a
*blobs.KeyErrorwith the bucket and the whole path;errors.Isfinds the store's sentinel in it. A commit whose outcome is unknown isblobs.ErrOutcomeUnknown: the object may or may not be there, and its callerStats it before writing again.
Testing without waiting
A Manual store with a clock the test moves runs no background work, so expiry, the scrub and the removal of what uploads left happen when the test says:
now := time.Now()
store, err := tinystore.Open(ctx, t.TempDir(), tinystore.Options{Manual: true, Clock: func() time.Time { return now }})
objects, err := blobs.Open(ctx, store, blobs.Options{})
exports, err := blobs.OpenBucket(ctx, objects, "exports", blobs.DefaultTTL(24*time.Hour))
_, err = exports.Put(ctx, "notes.zip", bytes.NewReader(archive))
now = now.Add(25 * time.Hour)
_, found, err := exports.Open(ctx, "notes.zip") // false: expired
_, err = objects.Maintain(ctx) // removes it and its file, and scrubs a slice
Not in the first version
What design/blobs.md leaves for later: resuming an upload, packs for small files, totals kept for folders, checked ranges, sharing the bytes of equal uploads, compression, a folder's own objects and sub-folders, versions kept after a replace, a quota enforced in the commit, transactions over several calls, appends.
Documentation
¶
Overview ¶
Package blobs keeps an application's files in blobs/ inside a tinystore.Store: objects under keys that are paths, their rows in blobs.db, their bytes inline up to 16 KiB and in a file of their own above it.
objects, err := blobs.Open(ctx, store, blobs.Options{})
avatars, err := blobs.OpenBucket(ctx, objects, "avatars", blobs.MaxSize(20<<20))
obj, err := avatars.Of(user.ID).Put(ctx, "original", r.Body, blobs.ContentType("image/jpeg"))
photo, found, err := avatars.Of(user.ID).Open(ctx, "original") // Read, ReadAt, Seek, Close
An object appears at its commit, whole, or not at all: its bytes are synced and renamed into objects/ before the row that names them commits, and a file is never written or renamed again, so a reader keeps what it opened. Start with blobs.go and bucket.go; every other file holds one step:
blobs.go Open, Close: the directory, the file, admission and the gates' steps bucket.go OpenBucket, Of: a handle on a bucket or a folder of it options.go the engine's bounds, a bucket's options and a call's keys.go what a path is, and the range of a folder or a prefix object.go Object, Query, Page, Usage, KeyError, ETags and meta upload.go Put, Create, Write: bytes in memory, then in a file of uploads/ publish.go Commit, Abort: the file synced and renamed, the row committed commit.go the rows a write changes: its condition, its content and the names it takes read.go Open, Stat reader.go Reader: Read, ReadAt, Seek, and a whole read checked write.go Delete, Copy, Move scan.go Scan, All, Usage clear.go Clear: a folder deleted at once, or marked and deleted by Maintain ids.go content ids reserved in blocks, held until their uploads end files.go the files' names, their directories' syncs, and their removal recover.go what Open removes that a process that died left maintain.go Maintain: expiry, marks, files no key names, the settled mark scrub.go the scrub: every content read once a pass snapshot.go Snapshot: blobs.db copied and the files it names linked
Example (Avatars) ¶
An avatar's original and the sizes a page shows live in the user's folder. A job makes the sizes from whatever original is there when it runs, and deleting the account clears the folder.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"time"
"github.com/tinyshed/tinystore"
"github.com/tinyshed/tinystore/blobs"
)
// clock is the store's clock in the examples, moved rather than waited for
type clock struct{ now time.Time }
func (c *clock) Now() time.Time { return c.now }
func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) }
// exampleObjects opens blobs in a Manual store in a new directory, so that an
// example moves its clock instead of sleeping; done closes and removes it
func exampleObjects() (objects *blobs.Store, at *clock, done func()) {
ctx := context.Background()
directory, err := os.MkdirTemp("", "tinystore-blobs-")
check(err)
at = &clock{now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)}
store, err := tinystore.Open(ctx, directory, tinystore.Options{Manual: true, Clock: at.Now})
check(err)
objects, err = blobs.Open(ctx, store, blobs.Options{})
check(err)
return objects, at, func() {
_ = store.Close(ctx)
_ = os.RemoveAll(directory)
}
}
// check ends an example that meets an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
objects, _, done := exampleObjects()
defer done()
avatars, err := blobs.OpenBucket(ctx, objects, "avatars", blobs.MaxSize(20<<20))
check(err)
mine := avatars.Of(42)
original, err := mine.Put(ctx, "original", bytes.NewReader(make([]byte, 300_000)),
blobs.ContentType("image/jpeg"))
check(err)
for _, side := range []int{256, 64} { // what the resizing job writes
sized, createErr := mine.Create(ctx, fmt.Sprintf("%d.jpg", side), blobs.ContentType("image/jpeg"))
check(createErr)
_, createErr = sized.Write(make([]byte, side*side/4))
check(createErr)
_, createErr = sized.Commit(ctx)
check(createErr)
sized.Abort() // after Commit it does nothing
}
for object, err := range mine.All(ctx, blobs.Query{}) {
check(err)
fmt.Println(object.Key, object.Size, object.ContentType)
}
_, err = mine.Put(ctx, "huge", bytes.NewReader(make([]byte, 21<<20)))
fmt.Println(errors.Is(err, tinystore.ErrLimit), original.Size)
check(mine.Clear(ctx))
_, found, err := mine.Stat(ctx, "original")
check(err)
fmt.Println(found)
}
Output: 256.jpg 16384 image/jpeg 64.jpg 1024 image/jpeg original 300000 image/jpeg true 300000 false
Example (ChatAttachments) ¶
A file the composer uploads waits a day under pending/. Sending moves it to sent/, where it stays, and the user's folder answers a quota check by counting its objects' rows.
package main
import (
"context"
"fmt"
"os"
"strings"
"time"
"github.com/tinyshed/tinystore"
"github.com/tinyshed/tinystore/blobs"
)
// clock is the store's clock in the examples, moved rather than waited for
type clock struct{ now time.Time }
func (c *clock) Now() time.Time { return c.now }
func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) }
// exampleObjects opens blobs in a Manual store in a new directory, so that an
// example moves its clock instead of sleeping; done closes and removes it
func exampleObjects() (objects *blobs.Store, at *clock, done func()) {
ctx := context.Background()
directory, err := os.MkdirTemp("", "tinystore-blobs-")
check(err)
at = &clock{now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)}
store, err := tinystore.Open(ctx, directory, tinystore.Options{Manual: true, Clock: at.Now})
check(err)
objects, err = blobs.Open(ctx, store, blobs.Options{})
check(err)
return objects, at, func() {
_ = store.Close(ctx)
_ = os.RemoveAll(directory)
}
}
// check ends an example that meets an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
objects, clock, done := exampleObjects()
defer done()
files, err := blobs.OpenBucket(ctx, objects, "chat-files")
check(err)
mine := files.Of("users", 7)
for _, id := range []string{"u1", "u2"} {
_, err = mine.Put(ctx, "pending/"+id, strings.NewReader("a photo of "+id), blobs.TTL(24*time.Hour),
blobs.ContentType("image/jpeg"), blobs.Meta("name", id+".jpg"))
check(err)
}
sent, err := mine.Move(ctx, "pending/u1", "sent/u1") // sending: no expiry in a bucket without a default
check(err)
fmt.Println(sent.Key, sent.Meta["name"], sent.Expires.IsZero())
clock.advance(25 * time.Hour) // u2 was never sent
usage, err := mine.Usage(ctx)
check(err)
fmt.Println(usage.Objects, usage.Bytes)
page, err := mine.Scan(ctx, blobs.Query{Prefix: "sent/", Limit: 50}) // "your files"
check(err)
fmt.Println(len(page.Objects), page.Objects[0].Key)
}
Output: sent/u1 u1.jpg true 1 13 1 sent/u1
Example (Documents) ¶
A client that retries an upload sends If-None-Match: *, and the retry adds nothing. An editor saving sends the ETag it read, and a save over someone else's is refused.
package main
import (
"context"
"errors"
"fmt"
"os"
"strings"
"time"
"github.com/tinyshed/tinystore"
"github.com/tinyshed/tinystore/blobs"
)
// clock is the store's clock in the examples, moved rather than waited for
type clock struct{ now time.Time }
func (c *clock) Now() time.Time { return c.now }
func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) }
// exampleObjects opens blobs in a Manual store in a new directory, so that an
// example moves its clock instead of sleeping; done closes and removes it
func exampleObjects() (objects *blobs.Store, at *clock, done func()) {
ctx := context.Background()
directory, err := os.MkdirTemp("", "tinystore-blobs-")
check(err)
at = &clock{now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)}
store, err := tinystore.Open(ctx, directory, tinystore.Options{Manual: true, Clock: at.Now})
check(err)
objects, err = blobs.Open(ctx, store, blobs.Options{})
check(err)
return objects, at, func() {
_ = store.Close(ctx)
_ = os.RemoveAll(directory)
}
}
// check ends an example that meets an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
objects, _, done := exampleObjects()
defer done()
documents, err := blobs.OpenBucket(ctx, objects, "documents")
check(err)
first, err := documents.Put(ctx, "plans/q4.md", strings.NewReader("# Q4"), blobs.IfNoneMatch())
check(err)
_, err = documents.Put(ctx, "plans/q4.md", strings.NewReader("# Q4"), blobs.IfNoneMatch()) // the retry
fmt.Println(errors.Is(err, tinystore.ErrConflict))
mine, err := documents.Put(ctx, "plans/q4.md", strings.NewReader("# Q4\nship blobs"), blobs.IfMatch(first.ETag))
check(err)
_, err = documents.Put(ctx, "plans/q4.md", strings.NewReader("# Q4\nstale"), blobs.IfMatch(first.ETag))
fmt.Println(errors.Is(err, tinystore.ErrConflict), mine.ETag != first.ETag)
}
Output: true true true
Example (Export) ¶
A job streams a zip into an upload; the link works for a day, and the bytes leave with the object.
package main
import (
"archive/zip"
"context"
"fmt"
"os"
"time"
"github.com/tinyshed/tinystore"
"github.com/tinyshed/tinystore/blobs"
)
// clock is the store's clock in the examples, moved rather than waited for
type clock struct{ now time.Time }
func (c *clock) Now() time.Time { return c.now }
func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) }
// exampleObjects opens blobs in a Manual store in a new directory, so that an
// example moves its clock instead of sleeping; done closes and removes it
func exampleObjects() (objects *blobs.Store, at *clock, done func()) {
ctx := context.Background()
directory, err := os.MkdirTemp("", "tinystore-blobs-")
check(err)
at = &clock{now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)}
store, err := tinystore.Open(ctx, directory, tinystore.Options{Manual: true, Clock: at.Now})
check(err)
objects, err = blobs.Open(ctx, store, blobs.Options{})
check(err)
return objects, at, func() {
_ = store.Close(ctx)
_ = os.RemoveAll(directory)
}
}
// check ends an example that meets an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
objects, clock, done := exampleObjects()
defer done()
exports, err := blobs.OpenBucket(ctx, objects, "exports", blobs.DefaultTTL(24*time.Hour))
check(err)
out, err := exports.Of(42).Create(ctx, "e1.zip", blobs.ContentType("application/zip"),
blobs.Meta("name", "notes-2026-09-27.zip"))
check(err)
defer out.Abort()
archive := zip.NewWriter(out)
note, err := archive.Create("note-1.txt")
check(err)
_, err = note.Write([]byte("milk, bread, eggs, tea"))
check(err)
check(archive.Close())
export, err := out.Commit(ctx)
check(err)
fmt.Println(export.Meta["name"], export.Expires.Sub(export.Modified))
clock.advance(25 * time.Hour)
_, found, err := exports.Of(42).Open(ctx, "e1.zip")
check(err)
maintained, err := objects.Maintain(ctx)
check(err)
fmt.Println(found, maintained.Expired)
}
Output: notes-2026-09-27.zip 24h0m0s false 1
Example (Recordings) ¶
A camera writes one object a minute, so a crash loses the minute being written and never the minutes before it. A player seeks, and the server answers its range with http.ServeContent.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"time"
"github.com/tinyshed/tinystore"
"github.com/tinyshed/tinystore/blobs"
)
// clock is the store's clock in the examples, moved rather than waited for
type clock struct{ now time.Time }
func (c *clock) Now() time.Time { return c.now }
func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) }
// exampleObjects opens blobs in a Manual store in a new directory, so that an
// example moves its clock instead of sleeping; done closes and removes it
func exampleObjects() (objects *blobs.Store, at *clock, done func()) {
ctx := context.Background()
directory, err := os.MkdirTemp("", "tinystore-blobs-")
check(err)
at = &clock{now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)}
store, err := tinystore.Open(ctx, directory, tinystore.Options{Manual: true, Clock: at.Now})
check(err)
objects, err = blobs.Open(ctx, store, blobs.Options{})
check(err)
return objects, at, func() {
_ = store.Close(ctx)
_ = os.RemoveAll(directory)
}
}
// check ends an example that meets an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
objects, _, done := exampleObjects()
defer done()
recordings, err := blobs.OpenBucket(ctx, objects, "recordings", blobs.DefaultTTL(30*24*time.Hour))
check(err)
start := time.Date(2026, 9, 27, 9, 30, 0, 0, time.UTC)
for minute := range 3 {
name := start.Add(time.Duration(minute)*time.Minute).Format("2006-01-02/15-04") + ".mp4"
segment, createErr := recordings.Of("cam-1").Create(ctx, name, blobs.ContentType("video/mp4"))
check(createErr)
_, createErr = io.Copy(segment, strings.NewReader(strings.Repeat(fmt.Sprint(minute), 100_000)))
check(createErr)
_, createErr = segment.Commit(ctx)
check(createErr)
}
day, err := recordings.Of("cam-1").Scan(ctx, blobs.Query{Prefix: "2026-09-27/"})
check(err)
fmt.Println(len(day.Objects), day.Objects[2].Key)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
video, found, openErr := recordings.Of("cam-1").Open(r.Context(), strings.TrimPrefix(r.URL.Path, "/"))
if openErr != nil || !found {
http.NotFound(w, r)
return
}
defer video.Close()
w.Header().Set("ETag", video.ETag)
w.Header().Set("Content-Type", video.ContentType)
http.ServeContent(w, r, "", video.Modified, video)
}))
defer server.Close()
request, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL+"/2026-09-27/09-32.mp4", nil)
check(err)
request.Header.Set("Range", "bytes=99990-99999")
response, err := http.DefaultClient.Do(request)
check(err)
defer response.Body.Close()
tail, err := io.ReadAll(response.Body)
check(err)
fmt.Println(response.StatusCode, string(tail))
}
Output: 3 2026-09-27/09-32.mp4 206 2222222222
Index ¶
- Variables
- type Bucket
- func (b *Bucket) All(ctx context.Context, query Query) iter.Seq2[Object, error]
- func (b *Bucket) Clear(ctx context.Context) error
- func (b *Bucket) Copy(ctx context.Context, from, to string, options ...Option) (Object, error)
- func (b *Bucket) Create(ctx context.Context, key string, options ...Option) (*Upload, error)
- func (b *Bucket) Delete(ctx context.Context, key string, options ...Option) error
- func (b *Bucket) Move(ctx context.Context, from, to string, options ...Option) (Object, error)
- func (b *Bucket) Of(owners ...any) *Bucket
- func (b *Bucket) Open(ctx context.Context, key string) (*Reader, bool, error)
- func (b *Bucket) Put(ctx context.Context, key string, r io.Reader, options ...Option) (Object, error)
- func (b *Bucket) Scan(ctx context.Context, query Query) (Page, error)
- func (b *Bucket) Stat(ctx context.Context, key string) (Object, bool, error)
- func (b *Bucket) Usage(ctx context.Context) (Usage, error)
- type BucketOption
- type KeyError
- type Maintenance
- type Object
- type Option
- type Options
- type Page
- type Query
- type Reader
- type Store
- type Upload
- type Usage
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrOutcomeUnknown = sqlite.ErrOutcomeUnknown
ErrOutcomeUnknown is a write whose group's commit failed: the object may or may not be there, and its caller should Stat it before writing again.
Functions ¶
This section is empty.
Types ¶
type Bucket ¶
type Bucket struct {
// contains filtered or unexported fields
}
Bucket is a handle on the objects of one bucket, or of the folder of it that Of names; it may be used from any number of goroutines.
func OpenBucket ¶
func OpenBucket(ctx context.Context, objects *Store, name string, options ...BucketOption) (*Bucket, error)
OpenBucket opens the bucket name of blobs.db, creating it the first time. A bucket holds bytes of any kind, so it has no type; its options are the program's, and may change between runs.
func (*Bucket) All ¶
All is every key Scan finds, a page of Scan at a time. Each page is its own snapshot and none is held between pages, so a slow loop keeps no reader open, and an object written or deleted during the walk may or may not be met. An error ends the walk as its last element.
func (*Bucket) Clear ¶
Clear removes every object of this handle's folder and of the folders under it, at once. Up to 10,000 objects go in one transaction.
A larger folder is marked cleared at the file's revision, which hides its objects from every call the moment the mark commits; maintenance then deletes them 10,000 a transaction. An object written after the Clear is a new object and stays.
func (*Bucket) Copy ¶
Copy writes the object under from again under to, sharing its bytes: no byte is read or written, whatever its size. The copy is a new version with the source's type and meta unless the call gives others, and with the expiry a Put would give it.
Onto its own key a Copy changes those and keeps the bytes. It takes the options Put takes but Size, its conditions for to. An absent source is tinystore.ErrConflict.
func (*Bucket) Create ¶
Create begins an upload of key: bytes go in with Write, and the object appears at Commit. It takes the options Put takes. A condition is checked now, so that a doomed upload of gigabytes stops before its first byte, and again in the commit, which decides.
func (*Bucket) Delete ¶
Delete removes key and its bytes, once no other key names them. An absent key is not an error, except to IfMatch, which deletes only the version with that ETag.
func (*Bucket) Of ¶
Of is the folder of this handle that owners name, a segment each: a string or an integer, an integer by its decimal text, so that Of("users", 42) is the folder users/42/, which never meets users/4/. An owner holding / is tinystore.ErrInvalid, returned by every call of the handle. A folder exists while it holds objects; nothing creates or removes one.
func (*Bucket) Open ¶
Open returns a reader of the object under key as it is at this moment, and whether a live object is there.
A reader of a file holds the file open and nothing of the engine's, so it reads on after its key is deleted, replaced or expired, and after the store closes, until its own Close.
An inline object is checked before its first byte. A file is read from its first byte to its last in order, and checked before its last bytes are handed over; a range is not checked.
func (*Bucket) Put ¶
func (b *Bucket) Put(ctx context.Context, key string, r io.Reader, options ...Option) (Object, error)
Put stores what r yields under key, and returns once the object is durable: its bytes synced and its row committed. Whatever the key held is replaced whole at that moment. A *bytes.Reader, *bytes.Buffer or *strings.Reader says its own Size.
Memory does not follow the object's size: 16 KiB hold the bytes while they may stay inline, and past them the bytes go to a file as they arrive. A long stream may use 64 KiB when the store's memory is available immediately.
func (*Bucket) Scan ¶
Scan is a page of every key under this handle's folder, the keys of the folders under it included, from one snapshot, in the byte order of their paths: 10.jpg before 9.jpg. With Query.Prefix it is the keys that start with it. A page holds at most 1000 objects.
func (*Bucket) Stat ¶
Stat is the object under key without its bytes, and whether a live object is there.
func (*Bucket) Usage ¶
Usage is the objects under this handle's folder and their bytes, the folders under it included, counted from their rows in one snapshot; research's design/blobs.md has what that costs. An expired object and one a Clear hid are not counted. It is a check and not a limit: two uploads that each pass it can pass a quota together.
type BucketOption ¶
type BucketOption func(*bucketSettings)
BucketOption changes how a bucket treats its objects; none changes what its rows mean, so a program may change them between runs.
func DefaultTTL ¶
func DefaultTTL(d time.Duration) BucketOption
DefaultTTL is the expiry a write gives an object when it names none, d from the write.
func MaxSize ¶
func MaxSize(n int64) BucketOption
MaxSize bounds the bucket's objects: the upload that passes n bytes stops with tinystore.ErrLimit and leaves nothing. There is no bound unless it says.
type KeyError ¶
KeyError is a call refused because of one object: its bucket and its whole path, owners included. errors.Is finds the store's sentinel in it.
type Maintenance ¶
type Maintenance struct {
Expired int // objects past their expiry, removed
Cleared int // objects a marked Clear hid, removed
Removed int // files of contents no key names, removed
Aborted int // uploads whose context ended while no call held them
Scrubbed int64 // bytes the scrub read
Damaged int // contents the scrub found changed or missing
}
type Object ¶
type Object struct {
// Key is the object's path under the handle that returned it.
Key string
Size int64
// ETag is quoted, as an HTTP header carries it, and derives from the
// bytes, so two objects with the same bytes have one ETag. It is compared
// with ETags and nothing else.
ETag string
ContentType string
// Modified is the store's clock at the commit of this version.
Modified time.Time
// Expires is zero for an object that does not expire.
Expires time.Time
Meta map[string]string
}
Object is an object as the file knows it.
type Option ¶
type Option func(*callSettings)
Option changes one call. Put and Create take every option; Copy and Move all but Size; Delete only IfMatch. Any other is tinystore.ErrInvalid.
func ContentType ¶
ContentType is the object's type, kept as it is given, up to 256 bytes.
func IfMatch ¶
IfMatch writes or deletes only while the key holds a live object with this ETag, as an HTTP If-Match header spells it: a list of ETags, any of them, or * for any live object. Any other version, and an absent key, is tinystore.ErrConflict.
func IfNoneMatch ¶
func IfNoneMatch() Option
IfNoneMatch writes only while the key holds no live object, so that an upload retried after a timeout adds nothing.
func Meta ¶
Meta adds a string for serving the object, a file's original name or a width: a key is [a-z0-9][a-z0-9_-]{0,63}, and keys and values together are at most 2 KiB. A write that names Meta replaces all of it.
type Options ¶
type Options struct {
// KeepFree is the disk an upload leaves free, so that one upload cannot
// take the space every engine's commits need. It is 1 GiB when zero, and
// nothing is checked when it is negative.
KeepFree int64
}
Options changes how the engine keeps its files.
type Page ¶
Page is one page of a Scan, from one snapshot. More says the limit ended it before the folder did, and Next is the query that reads on.
type Query ¶
type Query struct {
// Prefix keeps only the keys that start with it.
Prefix string
// After keeps only the keys past it, when it is not empty.
After string
Limit int // objects a page returns: 100 when zero, at most 1000
}
Query asks Scan for a page of every key under a handle's folder, its sub-folders' included, in the byte order of their paths.
type Reader ¶
type Reader struct {
Object
// contains filtered or unexported fields
}
Reader is an object as Open found it: its fields, and its bytes through Read, ReadAt and Seek, so that http.ServeContent serves it with its ranges and conditions. A reader of a file holds the file open and nothing of the engine's; an inline object's reader holds its bytes. ReadAt may be called from several goroutines, Read and Seek from one.
func (*Reader) Close ¶
Close closes the reader's file and gives back the memory an inline object's bytes held; a second Close does nothing.
func (*Reader) Read ¶
Read reads on from where the last Read or Seek left off. A file read from its first byte to its last in order is checked: the Read that would hand over the last bytes of a file that does not hash to the object's SHA-256 is tinystore.ErrCorrupt instead, so a copy of a changed object fails before its end.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the engine: blobs.db, and the files beside it in blobs/.
func Open ¶
Open opens blobs/ inside the store: blobs.db and the files beside it. It removes what the uploads of a process that died left, and the store closes it and, unless it is Manual, runs its maintenance every minute.
func (*Store) Close ¶
Close aborts the uploads that have not committed, lets the work in flight finish, settles the ids no upload holds any more and closes blobs.db; a reader of a file reads on until its own Close. Cancellation stops waiting, not the cleanup. The store calls it: an application closes the store instead.
func (*Store) Maintain ¶
func (s *Store) Maintain(ctx context.Context) (Maintenance, error)
Maintain removes expired objects and the objects marked Clears hid, 10,000 a transaction and at most ten transactions of each a call, and removes the files no key names any more.
It also aborts the uploads whose context has ended, removes what uploads left behind, moves the settled mark up, and scrubs a slice of the contents. The store calls it every minute unless it is Manual.
func (*Store) Snapshot ¶
Snapshot copies blobs.db into dir while the engine keeps working, and links there every file the copy names. A link copies no byte and keeps the bytes after the engine removes its own name, so a 17 GB film costs the snapshot a directory entry.
Collection waits meanwhile, so that no file the copy names goes before it is linked. A file missing is left out and logged, and one the system will not link is copied.
type Upload ¶
type Upload struct {
// contains filtered or unexported fields
}
Upload is an object whose bytes arrive over time: Write takes them, Commit makes the object appear whole under its key, and Abort leaves nothing, so that defer upload.Abort() is always safe. An Upload is one goroutine's.
It lives as long as the context given to Create: a call after that context ended finds the upload aborted, and maintenance aborts one left without calls. The store's Close aborts it too. Used after either, or after Commit or Abort, it is the context's error or tinystore.ErrClosed.
func (*Upload) Abort ¶
func (u *Upload) Abort()
Abort ends the upload and leaves nothing; after Commit it does nothing.
func (*Upload) Commit ¶
Commit makes the object appear under its key, whole, and returns it once it is durable: its file synced and renamed into objects/ with its directory synced, then its row committed beside the writes of other goroutines. A stream shorter than its Size is tinystore.ErrInvalid, and a condition that no longer holds tinystore.ErrConflict; either leaves nothing.
Source Files