Documentation
¶
Overview ¶
Package blobs is the filesystem half of a library: its folder, and the papers/ folder inside it where PDFs live (ADR-011, ADR-012).
D4 named each PDF by its SHA-256. D18 replaced that with readable names, one folder per work — Piwowar-2018-The-state-of-OA--F7K2M9QXA/published.pdf — and moved the hash into the database, where it still deduplicates and still verifies. This package knows the names, the layout and the bytes; it knows nothing about SQL. What a file belongs to is internal/store's business, and reconciling the two is internal/library's.
Two rules hold everywhere here:
- The researcher owns the tree. fil writes new folders and files, and renames only what it named; it never moves, edits or deletes anything a person put there.
- Writes are atomic. A file is written under .filiation/tmp/, synced, checked and only then renamed into place, so a crash leaves at most a temporary file — never a bad PDF under a good name.
Index ¶
- Constants
- Variables
- func Base(rel string) string
- func Clean(rel string) string
- func Dir(rel string) string
- func FileVersion(name string) string
- func FolderID(name string) (string, bool)
- func FolderName(w model.Work) string
- func IsPDF(b []byte) bool
- type Entry
- type File
- type Folder
- func (f *Folder) Abs(rel string) string
- func (f *Folder) Hash(rel string) (File, error)
- func (f *Folder) Inbox() ([]Entry, error)
- func (f *Folder) MakeWorkFolder(name string) (string, error)
- func (f *Folder) MoveInto(rel, dir string) (string, error)
- func (f *Folder) Rel(abs string) (string, error)
- func (f *Folder) RenameFolder(rel, name string) (string, error)
- func (f *Folder) Root() string
- func (f *Folder) Stage(r io.Reader, opts WriteOptions) (*Staged, error)
- func (f *Folder) Walk() (Listing, error)
- func (f *Folder) Write(rel string, r io.Reader, opts WriteOptions) (File, error)
- func (f *Folder) WriteReport(text string) error
- type Listing
- type Staged
- type WorkFolder
- type WriteOptions
Constants ¶
const ( PapersDir = "papers" InboxDir = "Inbox" ExportsDir = "exports" )
The parts of a library folder (ADR-011).
const ( // LongPath is the length past which fil warns that a path is close to // Windows' 260-character limit, which some tools still enforce. LongPath = 240 )
const ReportName = "fil-report.txt"
ReportName is the note fil leaves in Inbox/ saying why each file still there was not filed, so a researcher who only looks at the folder still finds out.
Variables ¶
var ErrNotPDF = errors.New("not a PDF")
ErrNotPDF is returned when bytes meant to be a PDF are not one: a publisher's HTML landing page served under a .pdf link is the usual case.
var ErrTooLarge = errors.New("larger than the size limit")
ErrTooLarge is returned when bytes run past WriteOptions.MaxSize.
Functions ¶
func Clean ¶
Clean turns a path relative to the library folder into the form the database stores: forward slashes, no leading ./, and Unicode NFC. macOS stores accented names decomposed, so without NFC the same folder would compare unequal on different machines.
func FileVersion ¶
FileVersion maps a file name fil gave to the version of the paper it holds, in OpenAlex's vocabulary, or "" for any other name.
func FolderID ¶
FolderID returns the fil ID at the end of a folder's name, normalised, and whether there is one. It accepts any letter case, because a person may retype a name and some file systems do not keep case.
func FolderName ¶
FolderName is the name fil gives a work's folder: {FirstAuthorSurname}-{Year}-{ShortTitle}--{filID}.
Accents are folded to ASCII and everything but letters and digits is dropped, so no character Windows forbids can survive, and a name ending in a fil ID is never a reserved one like CON or NUL. A work with no author gives Unknown, with no year nd, and with no title — or one in a script with no ASCII letters — Untitled. The fil ID at the end keeps names unique on case-insensitive file systems and lets a scan find a folder the researcher moved.
Types ¶
type File ¶
type File struct {
Path string // relative to the library folder, as stored
SHA256 string // hex
Size int64
}
File is a file fil has written or read: where it is, and what is in it.
type Folder ¶
type Folder struct {
// contains filtered or unexported fields
}
Folder is a library folder on disk.
func Open ¶
Open prepares the library folder at root: it creates papers/, Inbox/, exports/ and the temporary directory if they are missing, and clears temporary files a crash left behind. It creates nothing else, and touches nothing the researcher put there.
func (*Folder) Inbox ¶
Inbox lists the PDFs in Inbox/, at any depth, oldest first by path. Hidden files and system litter are skipped, as in papers/.
func (*Folder) MakeWorkFolder ¶
MakeWorkFolder creates papers/<name> and returns its stored path. A folder already there is used as it is.
func (*Folder) MoveInto ¶
MoveInto moves the file at rel into the folder dir, keeping its name — or "name-2.pdf" and on if the name is taken: nothing is ever overwritten. It returns the new path. Both must be inside the library folder, so the move is a rename on one volume.
func (*Folder) RenameFolder ¶
RenameFolder moves a folder fil named to a new name in the same parent, and returns the new stored path. It refuses to overwrite.
func (*Folder) Stage ¶
Stage writes r to a temporary file in the library folder, syncs it and checks it. On any failure nothing is left behind.
func (*Folder) Walk ¶
Walk lists papers/. It reads directory entries only — no file is opened — so it stays well under a second for tens of thousands of folders.
Hidden files and folders, and the litter operating systems leave behind, are skipped. Symbolic links are not followed. Everything inside a work folder belongs to that work, at any depth; a work folder inside another work folder is treated as part of the outer one.
func (*Folder) Write ¶
Write stores r at rel atomically: Stage, then Commit. Any failure leaves rel untouched and no temporary file behind. Write refuses to replace a file that is already there — whatever is at rel is either fil's own earlier copy or the researcher's, and neither is for overwriting.
func (*Folder) WriteReport ¶
WriteReport replaces Inbox/fil-report.txt with text, or removes it when text is empty. It is fil's own file, so replacing it is allowed.
type Listing ¶
type Listing struct {
// WorkFolders are folders whose name ends in a fil ID, wherever they sit.
WorkFolders []WorkFolder
// Loose are files that are not inside a work folder: PDFs dropped in by
// the researcher, or the contents of a work folder whose fil ID was taken
// off its name.
Loose []Entry
// Folders are the researcher's own folders — every folder that is not a
// work folder or inside one — relative to the library folder.
Folders []string
}
Listing is what is in papers/ right now, before anyone has decided what any of it means.
type Staged ¶
Staged is a file written to the library's temporary directory, checked, and not yet in place. Its hash is known, so a caller can decide whether the bytes are wanted — a download fil already has, say — before they land anywhere a person would see them. Every Staged must end in Commit or Discard.
type WorkFolder ¶
type WorkFolder struct {
Path string // relative to the library folder
FilID string // as written in the name, normalised; may be an alias
Files []Entry
}
WorkFolder is a folder named for one work.
type WriteOptions ¶
type WriteOptions struct {
// PDF requires the bytes to start like a PDF.
PDF bool
// SHA256, when set, is the hash the bytes must have.
SHA256 string
// MaxSize stops a runaway download. Zero means no limit.
MaxSize int64
}
WriteOptions control Write.
Source Files