Documentation
¶
Overview ¶
Package recobj is the format of a record object: the one object an ingest batch becomes, as docs/reference/bucket-contract.md specifies it.
The body is newline-delimited JSON, zstd-compressed, and each line is
{"hash":"<hex sha256>","record":{...}}
where record is the profile's copy in its canonical form (RFC 8785 over the protobuf JSON mapping) and hash is the SHA-256 of exactly that encoding. The object's own SHA-256 and its record count are metadata, so that a reader needs a listing and a HEAD and never a body.
The writer builds objects here and every reader takes them apart here, so the format has one definition in the code as it has one in the contract.
Index ¶
- Constants
- func CheckMetadata(meta map[string]string, stored []byte, lines int) []string
- func Encode(lines [][]byte) (body []byte, metadata map[string]string)
- func EncodeLine(canonical []byte) []byte
- func Hash(canonical []byte) string
- func Metadata(stored []byte, count int) map[string]string
- func SHA256(stored []byte) string
- type Line
Constants ¶
const ( ContentType = "application/x-ndjson" Encoding = "zstd" )
ContentType and Encoding are what an object is stored as.
Variables ¶
This section is empty.
Functions ¶
func CheckMetadata ¶
CheckMetadata compares what an object's metadata says with the object: the format, the sha256 of the stored bytes and the number of records. Every disagreement is returned, so that a report can say all of them.
func EncodeLine ¶
EncodeLine makes the line for a record already in canonical form. The bytes are embedded as they are, so that the hash is over what is on the line.
Types ¶
type Line ¶
type Line struct {
// Hash is what the line claims for its record.
Hash string `json:"hash"`
// Record is the record as written, canonical.
Record json.RawMessage `json:"record"`
}
Line is one record of an object, as the object has it.
func Decode ¶
Decode decompresses an object and reads its lines. It does not check any hash: Verify does, and a reader that only wants the records does not pay for it.
Source Files