Documentation
¶
Overview ¶
Package attachments keeps the files attached to a task (Task 20428).
A task's transcript says what an agent did. An attachment is what it made for a person to look at: a report, a chart, a screenshot, a data export, a log. The transcript is not the place for any of those — a PNG does not survive being printed — and the working tree is not either: it is the repository's, it is often a worktree that is removed when the task ends, and on an isolating executor it is on another machine.
How a file gets attached ¶
A run attaches files by copying them into its task's outbox,
<store>/outbox/<task id>/
whose absolute path the agent is given in its prompt and in $CLOOP_TASK_ATTACHMENTS (PrepareOutbox). When the attempt ends the orchestrator moves every regular file it finds there into the store and records it (Ingest). A person attaches one by hand with `cloop task attachments add` (Add), and a remote run's files arrive in its result and are stored by the hub's merge (Put).
Where the bytes live ¶
In the store beside the project's database, statedb.AttachmentStoreDir: <project>/.cloop/attachments/<id>, where the id is 128 random bits in hex. Nothing a run chose ever names a file on disk — the name it attached under is a label in the database row, nothing more.
The trust boundary ¶
On a shared-filesystem executor the store is written from inside the sandbox, and the hub — on the host, often as root — reads it to serve viewers and writes it when a remote run's files come back. So every operation here goes through statedb.OpenAttachmentStore's os.Root, bound to the store directory itself: a store swapped for a link redirects nothing. A file read must be a regular file, reached without following a link, and the same file after it is opened as before (Open). The store also carries a .gitignore that ignores everything in it, so a project that does not ignore .cloop/ cannot commit a run's attachments by accident.
Index ¶
- Constants
- Variables
- func Add(db *statedb.DB, dbPath string, taskID int, path string, meta Meta) (statedb.AttachmentRow, error)
- func BaseName(name string) string
- func CleanName(name string) string
- func CloseOutbox(dbPath string, taskID int)
- func Delete(db *statedb.DB, taskID int, id, actor, via string) (statedb.AttachmentRow, error)
- func DetectType(name string, head []byte) string
- func Env(taskID int, outbox string) []string
- func HumanBytes(n int64) string
- func InlineType(recorded string, head []byte) (string, bool)
- func Open(dbPath string, row statedb.AttachmentRow) (*os.File, os.FileInfo, error)
- func OutboxDir(dbPath string, taskID int) string
- func Outboxes(dbPath string) []int
- func PrepareOutbox(dbPath string, taskID int) (string, error)
- func Previewable(recorded string) bool
- func PromptSection(taskID int, outbox string) string
- func Put(db *statedb.DB, dbPath string, taskID int, meta Meta, src io.Reader, ...) (statedb.AttachmentRow, error)
- func Record(db *statedb.DB, taskID int, meta Meta, size int64, ...) (statedb.AttachmentRow, error)
- func RemoveOutbox(dbPath string, taskID int)
- func Resolve(rows []statedb.AttachmentRow, ref string) (statedb.AttachmentRow, error)
- func Room(u statedb.AttachmentUsage, size int64) string
- func StoreDir(dbPath string) string
- func Summary(taskID int, rep Report) string
- func Sweep(db *statedb.DB, dbPath string, minAge time.Duration) (int, error)
- func TypeByName(name string) string
- type IngestOptions
- type Meta
- type Refusal
- type Report
Constants ¶
const ( // MaxFileBytes bounds one file. MaxFileBytes int64 = 32 << 20 // MaxTaskFiles and MaxTaskBytes bound one task's attachments together. MaxTaskFiles = 100 MaxTaskBytes int64 = 128 << 20 // MaxProjectBytes bounds everything one project's store holds: a run that // attaches a gigabyte of plots every task would otherwise fill the disk // the hub and every other project share. MaxProjectBytes int64 = 1 << 30 )
Limits. Every attach is checked against all of them; a file over one is refused and named, never truncated.
const ( // EnvOutbox names the absolute path of the task's outbox. EnvOutbox = "CLOOP_TASK_ATTACHMENTS" // EnvTaskID names the task the agent is working on. EnvTaskID = "CLOOP_TASK_ID" )
Environment variables a run's agent is given.
const OutboxDirName = statedb.AttachmentOutboxDir
OutboxDirName is the store's subdirectory that holds the outboxes.
Variables ¶
var ( // ErrTooLarge reports a file over MaxFileBytes. ErrTooLarge = errors.New("attachments: file is over the size limit") // ErrQuota reports an attach the task's or the project's limits leave no // room for. ErrQuota = errors.New("attachments: no room left for the file") ErrUnavailable = errors.New("attachments: the file is not in the store") // ErrNotRegular reports something in the store or the outbox that is not // a plain file: a link, a directory, a FIFO. ErrNotRegular = errors.New("attachments: not a regular file") )
Errors.
var ErrAmbiguous = errors.New("attachments: more than one attachment matches")
ErrAmbiguous reports a reference that matches more than one attachment.
Functions ¶
func Add ¶
func Add(db *statedb.DB, dbPath string, taskID int, path string, meta Meta) (statedb.AttachmentRow, error)
Add attaches the file at path to a task by hand, refusing it when it is not a regular file or there is no room for it.
func BaseName ¶
BaseName is the last component of an attachment's name: what a download is saved as.
func CleanName ¶
CleanName turns the path a file had in an outbox, or the name it was given on a command line, into the name it is attached under: slash-separated, no empty, "." or ".." components, no control characters, valid UTF-8, at most statedb.MaxAttachmentNameBytes bytes. The name is a label — it never names a file on disk — but it is shown in the dashboard, in a terminal and in a download's Content-Disposition, so it is kept printable.
Format characters go too: a right-to-left override (U+202E) placed before "txt.exe" is how an executable is made to read as a text file.
A name over the limit keeps its end, where the file name and its extension are.
func CloseOutbox ¶
CloseOutbox removes a task's outbox once an attempt is over, when it is empty. One that still holds a file an ingest could not take yet is kept for the next attempt to try again.
func Delete ¶
Delete removes one attachment of a task at a person's request — its row, then its file — and records who did it: an audit row, and a line in the project's journal beside the task's history. db must be the project's own handle (statedb.DB.AsProject), where a task's audit rows belong. via names the surface that asked: "dashboard", "cloop task attachments remove".
func DetectType ¶
DetectType returns the media type recorded for a file named name whose content starts with head.
func Env ¶
Env is what an agent working on a task is given: where its outbox is, and which task it is working on.
func InlineType ¶
InlineType returns the Content-Type to display an attachment under, and whether it may be displayed at all, from its recorded type and the first bytes of what is in the store now. The bytes are looked at again because the store is in a workload's reach: a recorded type is only a claim.
func Open ¶
Open opens an attachment's stored bytes for reading.
The file is reached through the store's os.Root, so no link can lead out of it, and it must be a regular file both when it is looked at and once it is open — so a link, or a FIFO, or a file swapped for one in between, is refused rather than read. A file whose size is not the one recorded is refused too: the store was changed behind the database's back.
func PrepareOutbox ¶
PrepareOutbox makes sure the task's outbox exists, empty or not, and returns its absolute path: what the agent is told to copy files into.
func Previewable ¶
Previewable reports whether the dashboard should offer to show a file of the recorded type, before it fetches a byte of it. InlineType has the last word.
func PromptSection ¶
PromptSection is what a task's prompt says about its outbox. Short: it is part of every task's prompt.
func Put ¶
func Put(db *statedb.DB, dbPath string, taskID int, meta Meta, src io.Reader, limit int64, scrub *redact.Set) (statedb.AttachmentRow, error)
Put stores the bytes src yields as a new attachment of the task, records it and returns the row. limit bounds what is read from src; scrub, when set, removes leased credentials from the bytes on their way into the store, and the size and digest recorded are of what was stored.
The caller checks the task's and project's room first (Room); Put checks only the file itself, against limit.
func Record ¶
func Record(db *statedb.DB, taskID int, meta Meta, size int64, sha256Hex, unavailable string) (statedb.AttachmentRow, error)
Record records an attachment whose bytes are not in this store — a remote run's file that its result had no room for — with why they are not. Its size and digest are what the executor reported.
func RemoveOutbox ¶
RemoveOutbox removes a task's outbox and whatever is in it: the outbox of a task the plan no longer has.
func Resolve ¶
func Resolve(rows []statedb.AttachmentRow, ref string) (statedb.AttachmentRow, error)
Resolve finds the attachment ref names among a task's rows: its full id, an unambiguous id prefix of at least six characters, or its exact name.
func Room ¶
func Room(u statedb.AttachmentUsage, size int64) string
Room reports why the task cannot take one more file of size bytes, or "" when it can.
func Sweep ¶
Sweep removes what the store holds that no row names: files whose rows were deleted by something that could not remove them, and copies an attach abandoned half-way. Only what is older than minAge is removed, so a file an attach in another process has stored but not yet recorded is left alone.
func TypeByName ¶
TypeByName is the type a file's name claims, for an attachment whose bytes cannot be looked at: one a remote run left on its executor. Only a label — nothing is ever served on its strength.
Types ¶
type IngestOptions ¶
type IngestOptions struct {
// RunID is the execution the files are attributed to.
RunID string
// Scrub removes the credentials leased to the run from the files on their
// way into the store, as the provider's redaction does for its transcript.
Scrub *redact.Set
}
IngestOptions describe the attempt whose outbox is being emptied.
type Meta ¶
type Meta struct {
Name string
Source string
RunID string
Note string
AddedBy string
CreatedAt time.Time
}
Meta is what an attach records besides the bytes.
type Report ¶
type Report struct {
Added []statedb.AttachmentRow
Refused []Refusal
}
Report is what one ingest did.
func Ingest ¶
Ingest moves the files a run left in a task's outbox into the store and records them. Every regular file at any depth is attached under its path inside the outbox; links, devices and FIFOs are not followed or read, only named in the report, as is every file the limits leave no room for.
The outbox is emptied of everything it handled, attached or refused, and left in place itself: an attempt that is still running may put more in it. A file that failed for a reason that may pass — a full disk, a busy database — stays, and is tried again by the next ingest. An outbox that does not exist is not an error: most attempts attach nothing.
Source Files