Documentation
¶
Overview ¶
Package portable reads and writes the vendor-neutral nself portable export bundle (contract:cli.portable-export v1, P7-ADOPT-23).
A bundle is a directory: manifest.json plus the member files it lists. The Writer produces a deterministic manifest; the Reader treats the directory as untrusted input and refuses an unknown major version (E515) or any integrity failure (E516): a missing, changed, truncated, oversized, unlisted, duplicated or unsafely named member, a member that is no longer the file that was verified, and any symlink or hard link. Member names are Unicode NFC and unique under case folding; storage objects are content-addressed (StorageMember).
Layering: L1; imports only internal/errs, the standard library and golang.org/x/text (NFC and case folding).
Index ¶
- Constants
- Variables
- func CheckMember(rel string) error
- func MarshalManifest(m Manifest) ([]byte, error)
- func StorageMember(bucket, key string) (string, error)
- type Auth
- type Bucket
- type DB
- type Exemption
- type File
- type Hasura
- type HasuraSkipped
- type Limits
- type Manifest
- type Object
- type PostgresInfo
- type Producer
- type Reader
- type SourceCount
- type Storage
- type Table
- type Writer
Constants ¶
const ( // SHA256Pattern matches a lowercase hex SHA-256. SHA256Pattern = `^[0-9a-f]{64}$` // HashPattern matches a table hash in canonical form: a decimal integer // with no sign on zero, no leading zeros and no "+" ("0", "7", "-12"). // Hashes are compared as strings, so one number has one spelling. HashPattern = `^(0|-?[1-9][0-9]*)$` // VersionPattern matches a v1 schema_version ("1", "1.2"): major 1 only. VersionPattern = `^1(\.[0-9]+)*$` )
Patterns shared with the generated JSON schema.
const Format = "nself-portable-export"
Format is the manifest "_format" marker.
const MajorVersion = 1
MajorVersion is the only major this package reads and writes.
const ManifestName = "manifest.json"
ManifestName is the manifest file name at the bundle root.
const SchemaVersion = "1"
SchemaVersion is the manifest version the Writer emits. The Reader accepts any "1" or "1.<n>" and refuses every other major (E515).
const StorageMemberPattern = `^storage/objects/[0-9a-f]{64}$`
StorageMemberPattern matches a storage member path.
const StorageMemberPrefix = "storage/objects/"
StorageMemberPrefix is the directory of every stored object's bytes.
Variables ¶
var ( // ErrUnknownMajor: the manifest is not a v1 portable export (E515). ErrUnknownMajor = errors.New("unknown portable bundle format or major version") // ErrChecksum: a member's SHA-256 differs from the manifest (E516). ErrChecksum = errors.New("bundle file checksum mismatch") // ErrMissing: a listed member is absent (E516). ErrMissing = errors.New("bundle file missing") // ErrSize: a member's length differs from the manifest, or a size limit // was exceeded (E516). ErrSize = errors.New("bundle file size mismatch or limit exceeded") // ErrDuplicate: two members share a path (E516). ErrDuplicate = errors.New("duplicate bundle member") // ErrUnlisted: an entry exists in the bundle directory that the manifest // does not list (E516). ErrUnlisted = errors.New("bundle entry not listed in the manifest") // ErrLink: a member is a symlink, a hard link or not a regular file (E516). ErrLink = errors.New("bundle member is a link or not a regular file") // ErrChanged: a member is not the file that was verified at Open (another // file, or the same file with a different size or modification time) (E516). ErrChanged = errors.New("bundle file changed since it was verified") // ErrManifest: manifest.json is unreadable or fails validation (E516 for a // v1 manifest, E515 when it is not a v1 manifest at all). ErrManifest = errors.New("invalid bundle manifest") )
Sentinels wrapped inside the E515/E516 errors, for errors.Is.
var ( // Sources are the producer.source values. Sources = []string{"nself", "postgres", "nhost", "hasura-cloud", "supabase", "firebase", "appwrite"} // ExemptionKinds are the exemptions[].kind values. ExemptionKinds = []string{"table", "user", "object", "metadata"} // SourceCountKinds are the source_counts[].kind values. SourceCountKinds = []string{"table", "users", "objects"} )
Enumerations of the contract, used by the generated schema.
var DefaultLimits = Limits{
MaxManifestBytes: 64 << 20,
MaxFiles: 1_000_000,
MaxFileBytes: 1 << 40,
MaxTotalBytes: 8 << 40,
}
DefaultLimits are generous for real exports and small enough that a forged manifest cannot make a reader allocate or hash without bound. Reading is always streamed and capped at the declared length, so a member can never inflate past what the manifest declares.
var ErrUnsafePath = errors.New("unsafe bundle member path")
ErrUnsafePath is wrapped by CheckMember failures.
Functions ¶
func CheckMember ¶
CheckMember reports whether rel is a safe member path: relative, slash separated, made only of plain segments.
Purpose: a bundle is untrusted. A listed path must never leave the bundle directory on any platform, so the check is the strictest common subset. Refused: empty or over-long names; invalid UTF-8; control characters and NUL; a backslash (Windows separator); a leading slash; empty, "." and ".." segments (so no "../" and no "//"); ":" in any segment (Windows drive letters "C:", alternate data streams); a segment ending in "." or space; Windows device names (CON, NUL, COM1, LPT1 and the like, with any extension); a name that is not in Unicode NFC (so one name has one spelling, whatever the file system normalises). Outputs: nil, or an error wrapping ErrUnsafePath that says why (never more than the name, which is not a secret).
func MarshalManifest ¶
MarshalManifest returns the canonical manifest.json bytes of m (m should already be sorted).
func StorageMember ¶
StorageMember returns the member path of an object's bytes: storage/objects/<hex sha256(bucket + "\x00" + key)>.
Purpose: object keys are arbitrary and case- and Unicode-sensitive; file systems are not. Naming the file by a digest gives every (bucket, key) pair its own member on every platform, so keys like Readme.md and README.md, or the NFC and NFD spellings of one name, export on a case-insensitive or normalising file system as they do on S3. The real bucket and key live in the manifest (storage.objects[]); a reader maps key to member with this function and never parses the file name. Outputs: the member path, or an error wrapping ErrUnsafePath when the bucket is empty or holds NUL (it would make the digest input ambiguous), or the key is empty or not UTF-8.
Types ¶
type Auth ¶
type Auth struct {
Users int `json:"users"`
HashAlgorithms map[string]int `json:"hash_algorithms"`
ResetRequired []string `json:"reset_required"`
}
Auth summarises auth/users.jsonl.
type Bucket ¶
type Bucket struct {
Name string `json:"name"`
Objects int `json:"objects"`
Bytes int64 `json:"bytes"`
}
Bucket is one storage bucket.
type Exemption ¶
type Exemption struct {
// Kind is table, user, object or metadata.
Kind string `json:"kind"`
ID string `json:"id"`
Reason string `json:"reason"`
}
Exemption records one item deliberately left out, with the reason.
type File ¶
type File struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Bytes int64 `json:"bytes"`
}
File is one bundle member: path relative to the bundle root, with the SHA-256 (lowercase hex) and length of its exact bytes. manifest.json is not listed.
type Hasura ¶
type Hasura struct {
Tables int `json:"tables"`
Relationships int `json:"relationships"`
Permissions int `json:"permissions"`
Functions int `json:"functions"`
Skipped HasuraSkipped `json:"skipped"`
}
Hasura counts the exported metadata and what was skipped.
type HasuraSkipped ¶
type HasuraSkipped struct {
RemoteSchemas int `json:"remote_schemas"`
Actions int `json:"actions"`
}
HasuraSkipped counts metadata kinds the bundle does not carry.
type Limits ¶
type Limits struct {
// MaxManifestBytes caps manifest.json.
MaxManifestBytes int64
// MaxFiles caps the number of listed members (and of directory entries).
MaxFiles int
// MaxFileBytes caps one member's declared length.
MaxFileBytes int64
// MaxTotalBytes caps the sum of declared member lengths.
MaxTotalBytes int64
}
Limits bounds what Open accepts from an untrusted bundle. A zero field takes its DefaultLimits value.
type Manifest ¶
type Manifest struct {
Format string `json:"_format"`
SchemaVersion string `json:"schema_version"`
CreatedAt string `json:"created_at"`
Producer Producer `json:"producer"`
Postgres PostgresInfo `json:"postgres"`
DB DB `json:"db"`
Auth Auth `json:"auth"`
Storage Storage `json:"storage"`
Hasura Hasura `json:"hasura"`
Exemptions []Exemption `json:"exemptions"`
SourceCounts []SourceCount `json:"source_counts"`
Compat []string `json:"compat"`
Files []File `json:"files"`
}
Manifest is manifest.json, contract:cli.portable-export v1. The JSON schema schemas/portable-export.v1.schema.json is generated from these types by tools/schemagen.
Purpose: one definition of a bundle shared by the importer, the exporter and every source. Constraints: no field may carry a secret value (config lists key names only, in config/keys.txt). Arrays are sorted by Sort; maps marshal in key order. Only CreatedAt is time-dependent.
func (*Manifest) Sort ¶
func (m *Manifest) Sort()
Sort puts every array into its canonical order and replaces nil slices by empty ones, so the marshalled manifest is the same for equal content.
Purpose: Constitution 4.4 deterministic output (sorted keys and arrays). Constraints: PK keeps the key's column order (it is data, not a set); maps need no work because encoding/json marshals them in key order.
func (*Manifest) Validate ¶
Validate checks the manifest against the v1 rules that the generated JSON schema cannot express alone: marker and version, RFC 3339 created_at, lowercase hex SHA-256 values, canonical table hashes, non-negative counts, unique tables, safe NFC member paths that are unique under NFC and case folding (so two members never land on one file of a case-insensitive or normalising file system), and storage objects that each name their own listed file with the same SHA-256 and length, with (bucket, key) unique and no file under storage/objects/ left without an object.
type Object ¶
type Object struct {
Bucket string `json:"bucket"`
Key string `json:"key"`
Member string `json:"member"`
SHA256 string `json:"sha256"`
Bytes int64 `json:"bytes"`
}
Object is one stored object. Member is StorageMember(Bucket, Key), the bundle file that holds its bytes; SHA256 and Bytes equal that file's entry in files[]. (Bucket, Key) is unique and compared byte for byte.
type PostgresInfo ¶
PostgresInfo describes the server the data was read from.
type Producer ¶
type Producer struct {
Tool string `json:"tool"`
Version string `json:"version"`
// Source is nself, postgres, nhost, hasura-cloud, supabase, firebase or appwrite.
Source string `json:"source"`
}
Producer names the tool that wrote the bundle and the kind of source.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is an opened, fully verified bundle.
func OpenWithLimits ¶
OpenWithLimits is Open with explicit limits.
Purpose: the bundle is untrusted input. Open refuses it unless every check passes, in this order: manifest.json is a regular file within the size limit; it is a v1 portable export (unknown format or major: E515); it parses and validates (E516); member count and sizes are within the limits; every member path is safe and unique; no entry exists that the manifest does not list; every member is a regular, singly linked file reached without a symlink, with the declared length and SHA-256 (E516, naming the file). Outputs: a Reader whose Manifest and members are verified at open time. Constraints: never prints bundle content; messages name paths only.
func (*Reader) Open ¶
func (r *Reader) Open(rel string) (io.ReadCloser, error)
Open returns a reader for a listed member. It opens through OpenFile's checks and re-checks length and SHA-256 as it streams: reading to EOF yields E516 instead of io.EOF if the bytes changed since Open.
func (*Reader) OpenFile ¶
OpenFile opens a listed member and returns the descriptor, for tools that need a file rather than a stream (pg_restore reads standard input, or a /dev/fd path of this descriptor).
Purpose: a path handed to another process can be swapped between the check and the use. This returns an open file instead, after re-checking that the member is still the file that was verified at Open: no symlink on the way (O_NOFOLLOW on the last component where the platform has it), one hard link, and the same device and inode (Windows: volume and file index), size and modification time as the snapshot taken during verification. Any difference is refused with E516 (ErrChanged, or ErrLink for a link). Constraints: the bytes are not hashed again; Open streams them with the length and SHA-256 check. Callers must close the file.
type SourceCount ¶
type SourceCount struct {
// Kind is table, users or objects.
Kind string `json:"kind"`
ID string `json:"id"`
Count int64 `json:"count"`
Method string `json:"method"`
}
SourceCount is the source's own count of one kind, with how it was measured.
type Table ¶
type Table struct {
Schema string `json:"schema"`
Name string `json:"name"`
Rows int64 `json:"rows"`
Hash string `json:"hash"`
PK []string `json:"pk"`
}
Table is one dumped table; (Schema, Name) is unique. Hash is the canonical decimal string from internal/postgres/tablehash; PK keeps the key's column order.
type Writer ¶
type Writer struct {
// Now supplies created_at when the manifest leaves it empty; tests pin it.
Now func() time.Time
// contains filtered or unexported fields
}
Writer builds a bundle directory. It is not safe for concurrent use.
Purpose: one place that creates bundle files with the right modes and records each file's SHA-256 and length while it streams, so the manifest lists exactly the bytes that were written. Constraints: the bundle directory is 0700 and every file 0600 (EPIC D9). Files are created exclusively, so an existing file or a symlink in the way is an error, never followed. Member paths pass CheckMember.
func NewWriter ¶
NewWriter creates dir (0700) and returns a Writer for it. The directory must not exist, or must be empty; an existing non-empty directory is refused so a bundle never mixes with earlier content.
func (*Writer) Finish ¶
Finish completes m, sorts it, validates it and writes manifest.json (0600). Format and SchemaVersion are set when empty and must be the v1 values otherwise; CreatedAt defaults to Now() in UTC; Files is replaced by the members written through WriteFile. The manifest bytes are canonical: sorted arrays, sorted map keys, two-space indent, no HTML escaping, one trailing newline; the same input gives the same bytes.
func (*Writer) WriteFile ¶
WriteFile streams r into the member rel and records its SHA-256 and length. rel must pass CheckMember, must not be manifest.json, and must not repeat an earlier member (compared under NFC and case folding) or collide with a member that is a file where this one needs a directory, or the reverse. On a copy error the partial file is removed.
Source Files