README
¶
Wake signed log
The self hosted signed log: an append only, hash chained tlog-tiles log built on Trillian Tessera, with checkpoints signed by the log's key. The collector's wakelog exporter writes to it.
It stores the log on local disk for now. Writing to a customer's bucket with object lock comes when a design partner deploys it.
An entry appended again while the log still remembers it (Tessera's in memory antispam) lands once, so a request the collector retries after part of it was appended doesn't duplicate anything. The log forgets on restart, so an entry the collector's queue replays after a crash can still land twice, as an identical copy.
Generate a key pair with:
go run ./cmd/wake-keygen -name wake-log -out keys/log
Checking a log
Anyone can check a log without trusting Wake or whoever stores it. wake-log verify checks the checkpoint's signature, that every entry is in the signed tree, and every event's signature and payload. -entries also prints each entry as a JSON line.
go run ./cmd/wake-log verify -log ./data/log -log-key keys/log.pub \
-key keys/collector.pub -bundle identity/bundle.pem -signer proxy=spiffe://wake.local/proxy
-log is the log's directory, or an http(s) URL it's served from. -key names an event signer's public key (the collector's), -bundle a trust bundle that X.509-SVID signers (the proxy) must chain to, and -signer who may sign a capture, as <capture>=<SPIFFE ID> or a prefix ending in /. A bundle vouches for every workload in its trust domain, agents included, so an SVID signature only counts for a capture -signer lists it under, and -bundle needs -signer. All three repeat. In Go, wakelog.Verify does the same with a DirReader or Tessera's client.HTTPFetcher; both fall back to a full entry bundle when Tessera has deleted the partial one a checkpoint points to.
Each entry is compared with the collector's check when it appended it, and reported in one of three groups:
- Logged invalid, and still failing: evidence the collector recorded, such as an event signed by someone not allowed to. Reported; the log is fine.
- Logged valid, but failing now: the entry or its payload changed after it was logged, a payload file is missing, or this check trusts differently from the collector.
- Logged invalid, but passing now: this check trusts more than the collector did.
It exits 1 if the log doesn't hold together or any entry is in the last two groups.
Reading entries
Each entry holds an event's exact signed bytes, its signature and signer, the signer's certificate if it signs with an X.509-SVID (the proxy), the collector's check at append time, and the event's payload when it fits (the format is in docs/events.md). A payload too large for its entry is stored next to the tiles at payload/sha256/<digest>. ParseEntry reads an entry back without touching the signed bytes, Check checks the signature again, with the signer's public key or, for an SVID signer, its certificate against the SPIRE trust bundles as of when the collector received the event (CheckSignature says why it fails), ReadPayload returns the payload checked against its signed digest, and Decode reads the event's fields:
entry, err := wakelog.ParseEntry(data)
ok := entry.Check(wakelog.Trust{Keys: keys, Bundles: bundles,
Signers: wakelog.Signers{"proxy": {"spiffe://wake.local/proxy"}}}) // don't rely on entry.Valid
payload, err := entry.ReadPayload(func(path string) ([]byte, error) {
return os.ReadFile(filepath.Join(logDir, path)) // or read the bucket
})
event, err := entry.Decode()
ReadPayload only calls the fetch function when the payload isn't in the entry. It returns ErrNoPayload for an event without one and ErrPayloadMismatch when the payload doesn't match the digest.
The collector writes entries with the same code (Encode and Log.PutPayload), so the format is defined in one place.
Documentation
¶
Overview ¶
Package wakelog is Wake's self hosted signed log: an append only, hash chained tlog-tiles log built on Trillian Tessera, with checkpoints signed by the log's key.
Anyone can check a log without trusting Wake: its checkpoint, its tree and every event's signature, with Verify or the wake-log command:
go run github.com/manikoa/wake/log/cmd/wake-log verify -log <dir or url> -log-key log.pub -key collector.pub
Index ¶
- Constants
- Variables
- func Encode(event []byte, f Fields) (entry []byte, inline bool, err error)
- func GenerateKey(name string) (privateKey, publicKey string, err error)
- func NewEntry(event []byte, signature, signer string, valid bool) ([]byte, error)
- func NewEntryWithPayload(event []byte, payload string, signature, signer string, valid bool) (entry []byte, inline bool, err error)
- func PayloadPath(digest string) (string, error)
- func VerifySVIDSignature(msg []byte, signature, signer, certificate string, bundles *x509.CertPool, ...) error
- func VerifySignature(msg []byte, signature, signer string, trusted map[string]note.Verifier) bool
- type Checked
- type Config
- type DirReader
- type Entry
- type Event
- type Fields
- type Log
- type Reader
- type Signers
- type Trust
Constants ¶
const Capture = "wake.capture"
Capture is the signed attribute naming what recorded an event: sdk, proxy or memory_adapter. Trust.Signers says who may sign each.
const MaxClockSkew = 5 * time.Minute
MaxClockSkew is how far a signer's clock may run ahead of the collector's, and how close to the edge of its validity an X.509-SVID may be. An SVID signature is checked as of when the collector received the event, give or take this.
const MaxEntrySize = tessera.DefaultEntrySizeLimit
MaxEntrySize is the largest entry the log holds, in bytes. tlog-tiles stores each entry's length in two bytes.
const MaxEventAge = 24 * time.Hour
MaxEventAge is how long before the collector received it an event signed with an X.509-SVID may have started. It bounds how far back a signer can date an event. A long call, such as an A2A task, starts well before the proxy signs it, so this is generous.
const PayloadSHA256 = "wake.payload.sha256"
PayloadSHA256 is the signed attribute that commits an event to its payload.
Variables ¶
var ( // ErrEntryTooLarge means an entry is over MaxEntrySize even without its payload. ErrEntryTooLarge = errors.New("entry too large for the log") // ErrNoPayload means an event carries no payload digest, so it has no payload. ErrNoPayload = errors.New("event has no payload") // ErrPayloadMismatch means a payload doesn't match its event's signed digest. ErrPayloadMismatch = errors.New("payload doesn't match the signed wake.payload.sha256") )
Functions ¶
func Encode ¶ added in v0.3.0
Encode encodes a log entry. event must be the exact bytes that were signed, with the payload's digest in them and the payload left out. The payload goes in the entry when the entry still fits in the log and the payload is valid UTF-8, which a JSON string needs to carry it byte for byte. Otherwise it's left out and inline is false: store it with Log.PutPayload before appending the entry. With no payload, inline is true.
func GenerateKey ¶
GenerateKey returns a new note format key pair. name identifies the signer, for example a SPIFFE id.
func NewEntry ¶
NewEntry encodes a log entry for an event with no payload. event must be the exact bytes that were signed.
func NewEntryWithPayload ¶ added in v0.2.0
func NewEntryWithPayload(event []byte, payload string, signature, signer string, valid bool) (entry []byte, inline bool, err error)
NewEntryWithPayload encodes a log entry for an event with a payload. See Encode.
func PayloadPath ¶ added in v0.2.0
PayloadPath returns where a payload with this hex SHA-256 is stored, relative to the log's root: payload/sha256/<digest>. It refuses anything but 64 lowercase hex digits, since the digest comes from an event and must never name another path.
func VerifySVIDSignature ¶ added in v0.3.0
func VerifySVIDSignature(msg []byte, signature, signer, certificate string, bundles *x509.CertPool, at time.Time) error
VerifySVIDSignature checks a signature made with an X.509-SVID's key, as the proxy signs (docs/proxy.md). certificate is base64 of the DER SVID followed by any intermediates. The SVID must chain to one of bundles as of at. It must be an end-entity certificate with exactly one spiffe:// URI SAN, equal to signer, and a P-256 key. signature is base64 ASN.1 DER ECDSA over the SHA-256 of msg.
It doesn't say whether signer may sign the event, or which time to check at. Use Entry.CheckSignature, which checks both.
Types ¶
type Checked ¶ added in v0.4.0
type Checked struct {
Origin string
Size uint64
// Unsigned are the indexes of entries whose event signature doesn't check under the
// trust given, or that can't be read as entries at all.
Unsigned []uint64
}
Checked is what Verify found in a log.
func Verify ¶ added in v0.4.0
func Verify(ctx context.Context, r Reader, logKey note.Verifier, trust Trust, each func(index uint64, e Entry, signed bool)) (Checked, error)
Verify checks a log without trusting whoever wrote or serves it: the checkpoint is signed by logKey, the entries hash to the checkpoint's root, and each entry's event is signed by a signer in trust. each, if not nil, sees every entry in order, with whether its signature checked. An error means the log itself doesn't hold together; entries that fail their signature check are listed in Checked.Unsigned instead.
type Config ¶
type Config struct {
// Dir is where the tiles, the checkpoint and payloads too large for their entries
// are written.
Dir string
// KeyFile holds the note format private key that signs checkpoints.
KeyFile string
// CheckpointInterval is how often a new checkpoint is published. Defaults to one second.
CheckpointInterval time.Duration
}
Config says where the log lives and which key signs its checkpoints.
type DirReader ¶ added in v0.4.0
type DirReader string
DirReader reads the log in a local directory.
func (DirReader) ReadCheckpoint ¶ added in v0.4.0
func (DirReader) ReadEntryBundle ¶ added in v0.4.0
ReadEntryBundle reads a bundle, falling back from a partial bundle to the full one: once a bundle fills, Tessera deletes its partial versions, which a reader holding an older checkpoint still asks for. The full bundle starts with the same entries.
type Entry ¶
type Entry struct {
// Event holds the signed bytes exactly as they were signed.
Event json.RawMessage `json:"event"`
// Payload is the event's payload when it fits in the entry. It isn't signed; the
// signed wake.payload.sha256 commits to it. When it's nil and the event has a
// digest, the payload is stored next to the log at PayloadPath. Read it with
// ReadPayload, which checks it either way.
Payload *string `json:"payload,omitempty"`
Signature string `json:"signature"`
Signer string `json:"signer"`
// Certificate is set for a signer that signs with an X.509-SVID, such as the proxy:
// base64 of the DER certificate, followed by any intermediates. Check verifies it
// against SPIRE trust bundles.
Certificate string `json:"certificate,omitempty"`
// Received is when the collector received the event, in Unix nanoseconds. It isn't in
// the event's signed bytes, since the proxy signs before the collector receives the
// event; the log's signed checkpoint covers it, so read entries through Verify. Check
// checks an SVID as of this time, as the collector did.
Received uint64 `json:"received_unix_nano,omitempty"`
// Valid is the collector's check at append time. Check again with Check; don't trust it.
Valid bool `json:"valid"`
}
Entry is one log entry: an event's exact signed bytes, its signature and signer, whether the collector found the signature valid when it appended the entry, and the event's payload when it fits. The format is defined in docs/events.md.
func ParseEntry ¶
ParseEntry decodes a log entry, keeping the event's signed bytes exactly as they are.
func (Entry) Check ¶ added in v0.3.0
Check reports whether the entry's signature is valid for its event bytes under trust. CheckSignature says why when it isn't.
func (Entry) CheckSignature ¶ added in v0.4.0
CheckSignature returns nil when the entry's signature is valid for its event bytes under trust, and otherwise why it isn't. The signer must be a trusted note format key or an X.509-SVID that chains to a trusted bundle, and t.Signers must allow it to sign the event's capture (see Trust). An SVID is checked as of when the collector received the event, give or take MaxClockSkew, and the event must have started no more than MaxEventAge before that, so a signer can't pick the time its certificate is checked at. The collector runs the same check as events arrive.
func (Entry) Decode ¶
Decode returns the event inside the entry. Numbers in attributes decode as json.Number.
func (Entry) ReadPayload ¶ added in v0.2.0
ReadPayload returns the event's payload, checked against the signed wake.payload.sha256. It takes the payload from the entry when it's there, and otherwise calls fetch with PayloadPath, a path relative to the log's root, for example the tiles' directory or bucket. It returns ErrNoPayload if the event has no digest and ErrPayloadMismatch if the payload doesn't match it.
ReadPayload doesn't check the signature. Check it with Verify, which covers the digest.
type Event ¶
type Event struct {
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
StartTimeUnixNano uint64 `json:"start_time_unix_nano"`
EndTimeUnixNano uint64 `json:"end_time_unix_nano"`
Attributes map[string]any `json:"attributes"`
}
Event is the content of an entry's signed bytes.
type Fields ¶ added in v0.3.0
type Fields struct {
// Payload is the event's payload, or nil if it has none.
Payload *string
Signature string
Signer string
Certificate string
// Received is when the collector received the event, in Unix nanoseconds.
Received uint64
Valid bool
}
Fields are what an entry holds besides the event's signed bytes.
type Log ¶
type Log struct {
// contains filtered or unexported fields
}
Log appends entries to a signed log on local disk.
func (*Log) Add ¶
Add appends entries and waits until each has a durable index in the log. An entry the log already holds from recently isn't appended again; its index is the first one.
func (*Log) PutPayload ¶ added in v0.2.0
PutPayload stores a payload too large for its entry at PayloadPath of its SHA-256, next to the tiles. Call it before appending the entry, so no entry points to a payload that was never written. Storing the same payload again writes nothing.
type Reader ¶ added in v0.4.0
type Reader interface {
ReadCheckpoint(ctx context.Context) ([]byte, error)
ReadEntryBundle(ctx context.Context, index uint64, p uint8) ([]byte, error)
}
Reader reads a signed log: DirReader for its directory, or Tessera's client.HTTPFetcher for wherever the directory is served, such as a bucket.
type Signers ¶ added in v0.4.0
Signers lists, by wake.capture, who may sign events of that capture: signer names, such as SPIFFE IDs, or path prefixes ending in "/" that match every ID below them. A SPIFFE ID never ends in "/", so the two can't be confused.
type Trust ¶ added in v0.3.0
type Trust struct {
// Keys are note format public keys by signer name, such as the collector's.
Keys map[string]note.Verifier
// Bundles are the SPIRE trust bundles that X.509-SVID signers, such as the proxy,
// must chain to.
Bundles *x509.CertPool
// Signers says who may sign each capture. A bundle vouches for every workload in its
// trust domain, agents included, so an SVID signature only counts when its signer is
// listed for the event's capture. A note format key is trusted by being in Keys, but
// once a capture is listed here, only its listed signers count for it, whatever they
// sign with.
Signers Signers
}
Trust says whose signatures to accept.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
wake-keygen
command
Command wake-keygen writes a note format key pair: <out>.key (private) and <out>.pub.
|
Command wake-keygen writes a note format key pair: <out>.key (private) and <out>.pub. |
|
wake-log
command
Command wake-log checks a Wake signed log without trusting whoever wrote or serves it: the checkpoint's signature, that every entry is in the signed tree, and each event's signature and payload.
|
Command wake-log checks a Wake signed log without trusting whoever wrote or serves it: the checkpoint's signature, that every entry is in the signed tree, and each event's signature and payload. |