Documentation
¶
Overview ¶
Package wiretrace reads and writes wire traces: the plaintext of every encrypted MTProto message plus request, connection and storage lifecycle events, as written by the traced Telegram Desktop build (gotd/tdesktop fork, branch "trace").
Format ¶
A run directory holds segments:
- events-NNNN.jsonl: one JSON object per event. Every segment starts with a "hdr" record (seq 0) with "v", "seg", "run", "app", "layer", "api_sha256", "mtproto_sha256", "wall_utc", "mono_t", "pid" and "flags". Other records have "seq", "t" (monotonic seconds), "th" (thread) and "ev" (kind), plus kind-specific fields. 64-bit ids are decimal strings, session ids are signed, constructor ids are "0x%08x". Events emitted while a response or an update was handled carry "cause":{"req","outer","type"}. With UI tracing, "hdr" has "ui":{"v","ids","raw_ids"}, ui.* and app.* records describe user actions and view state, and main-thread events emitted during or shortly after an action carry "ui":{"a","rel","dt"} (UIStamp, see package wiretrace/uiattr).
- blobs-NNNN.bin: "TDWT", u16 version 1, then records of u64 seq, u32 len, u32 crc32c and the bytes. Events reference them with "blob":{"seg","off","len","crc32c","sha256"}, where off points at the bytes after the record header. Small bodies may also be inlined as "b64".
Segments rotate at 64 MiB of events or 256 MiB of blobs. When the total size cap is reached the oldest segments are deleted, except segment 0. If the writer queue overflows, a "dropped" record with "count" is written.
"msg" events carry the body of one encrypted message (after decryption, before encryption) with the header fields (sdc, session, salt, msg_id, seqno, len) and a summary (type, method, wrap, inner, req_msg_id, result). Bodies of sensitive requests and responses are redacted in place, keeping lengths: [id]RedactedMarker[id][zeros...], listed in "redacted".
Older tdesktop trace builds wrote numbers with the process locale decimal separator ("t":12,345678); ParseEvent repairs such lines.
Usage ¶
run, err := wiretrace.Open(dir)
...
err = run.Events(func(ev *wiretrace.Event) error {
body, err := run.Blob(ev) // verified against crc32c and sha256
...
}, nil)
Package wiretrace/tljson decodes TL bodies with the schema of the traced build, wiretrace/decode turns "msg" events into logical messages, and cmd/wiretrace is the command line tool (decode, timeline, stats, diff). Writer produces the same format, for tests and other emitters.
Index ¶
- Constants
- Variables
- func BlobsName(segment int) string
- func CRC32C(data []byte) uint32
- func EventsName(segment int) string
- func PeerValue(kind string, id int64) uint64
- func RepairLegacyNumbers(line []byte) ([]byte, bool)
- func VerifyBlob(seq uint64, ref *BlobRef, header, body []byte) error
- type BlobRef
- type Cause
- type Event
- func (e *Event) Bool(key string) bool
- func (e *Event) Decode(key string, v any) bool
- func (e *Event) Float(key string) float64
- func (e *Event) Has(key string) bool
- func (e *Event) Header() (*Header, error)
- func (e *Event) ID(key string) uint64
- func (e *Event) Int(key string) int64
- func (e *Event) Msg() (*Msg, error)
- func (e *Event) Str(key string) string
- func (e *Event) Type(key string) uint32
- type Field
- type Header
- type HeaderUI
- type ID
- type IDKey
- type Inner
- type Msg
- type Run
- type SignedID
- type TypeID
- type UIStamp
- type Writer
- type WriterOptions
Constants ¶
const ( // Version is the format version written in "hdr" records and in the blob // file header. Version = 1 // BlobMagic starts every blobs-NNNN.bin file, followed by u16 Version. BlobMagic = "TDWT" // BlobFileHeaderSize is the size of the blob file header. BlobFileHeaderSize = 6 // BlobRecordHeaderSize is the size of a blob record header: // u64 seq, u32 len, u32 crc32c (little-endian). BlobRecordHeaderSize = 16 // RedactedMarker is written right after the constructor id of a redacted // TL object: [id][RedactedMarker][id][zeros...]. Lengths are kept. RedactedMarker uint32 = 0x7EDAC7ED )
Format constants of a trace run directory.
const ( KindHeader = "hdr" KindDropped = "dropped" KindMsg = "msg" KindReqNew = "req.new" KindReqMsg = "req.msg" KindReqResend = "req.resend" KindReqDone = "req.done" KindReqFail = "req.fail" KindReqCancel = "req.cancel" KindReqCaller = "req.caller" KindReqMigrate = "req.migrate" KindState = "state" KindSessionReset = "session.reset" KindSessionNewID = "session.new_id" KindSaltChange = "salt.change" KindKeySet = "key.set" KindKeyCreated = "key.created" KindKeyDestroy = "key.destroy_temp" KindConnStart = "conn.start" KindConnUp = "conn.up" KindConnDown = "conn.down" KindConnReason = "conn.reason" KindConnRestart = "conn.restart" KindConfigLoaded = "config.loaded" KindStorageWrite = "st.write" KindStorageRaw = "st.write.raw" KindStorageEnc = "st.encrypt" KindStorageCommit = "st.commit" KindStorageRemove = "st.remove" KindStorageRead = "st.read" KindStorageDec = "st.decrypt" )
Event kinds written by the tdesktop tracer.
const ( KindUIPress = "ui.press" KindUIClick = "ui.click" KindUIKey = "ui.key" KindUIKeyEnd = "ui.key.end" KindUIScroll = "ui.scroll" KindUIScrollEnd = "ui.scroll.end" KindUIType = "ui.type" KindUITypeEnd = "ui.type.end" KindUIShortcut = "ui.shortcut" KindUIMenu = "ui.menu" KindUIMarker = "ui.marker" KindUIChat = "ui.chat" KindUISection = "ui.section" KindUIBox = "ui.box" KindUIMediaOpen = "ui.media.open" KindUIMediaClose = "ui.media.close" KindUICall = "ui.call" KindUIAccount = "ui.account" KindUISession = "ui.session" KindUISearchChat = "ui.search.chat" KindUIFolder = "ui.folder" KindAppWindow = "app.window" KindAppState = "app.state" )
UI event kinds (P14). Input actions carry "a", the action id; semantic and view-state events carry "ui" (the action they came from) or "a" with "src":"app" when they have no input parent.
const ( // UIRelSync: emitted inside the input scope (exact). UIRelSync = "sync" // UIRelDeferred: emitted shortly (kDeferredMs, 150 ms) after the last // input scope closed, with no newer input in between. UIRelDeferred = "deferred" // UIRelAfter: emitted within kAfterMs (3 s) after the last input scope, // or inside a scope opened more than 500 ms ago (a nested event loop, // dt counts from the scope open). UIRelAfter = "after" )
Relations of a UIStamp to its action.
const ( DirOut = "out" DirIn = "in" )
Directions of "msg" events.
const ( PeerUser = "user" PeerChat = "chat" PeerChannel = "channel" )
Peer kinds of PeerValue.
const HashPrefix = "h:"
HashPrefix starts hashed UI ids: "h:<16 hex>".
const MaxLineSize = 64 << 20
MaxLineSize limits the size of one events line.
const UIKeyName = "ui.key"
UIKeyName is the file of the per-run HMAC key of UI ids in a run directory. It is never written into events.
Variables ¶
var ErrBlobMismatch = errors.New("blob mismatch")
ErrBlobMismatch is returned when a blob record does not match its reference.
Functions ¶
func EventsName ¶
EventsName returns the name of the events file of segment.
func PeerValue ¶
PeerValue returns the tdesktop PeerId.value of a bare id: the id with the type in bits 48+ (user 0, chat 1, channel 2).
func RepairLegacyNumbers ¶
RepairLegacyNumbers rewrites numbers like `:12,345678` (comma decimal separator, written by tdesktop trace builds before the locale fix) into `:12.345678` when they are followed by `,`, `}` or `]`.
Valid JSON never has `:<digits>,<digits>` followed by one of those characters outside of strings, so the repair is applied only to lines that failed to parse. Returns false if nothing was changed.
Types ¶
type BlobRef ¶
type BlobRef struct {
Seg int `json:"seg"`
Off int64 `json:"off"`
Len int `json:"len"`
CRC32C uint32 `json:"crc32c"`
SHA256 string `json:"sha256,omitempty"`
}
BlobRef references a body in a blobs-NNNN.bin segment.
Off is the offset of the body itself, after the 16-byte record header.
type Cause ¶
type Cause struct {
// Req is the request id of the response, 0 for updates.
Req int64 `json:"req"`
// Outer is the outer msg_id of the response.
Outer ID `json:"outer"`
// Type is the constructor id of the response.
Type TypeID `json:"type"`
}
Cause is the causal context of an event emitted while a response or an update was being handled.
type Event ¶
type Event struct {
Seq uint64
T float64
Thread string
Kind string
// Blob references the body of the event, if any.
Blob *BlobRef
// Inline is the decoded "b64" field, if the writer inlined the body.
Inline []byte
// Cause is the causal context, if any.
Cause *Cause
// UI is the UI action stamp, if any. It is never set for "hdr" records
// (their "ui" is HeaderUI).
UI *UIStamp
// Fields holds all fields of the record.
Fields map[string]json.RawMessage
// Segment is the index of the events segment the record was read from.
Segment int
// Line is the 1-based line number in that segment.
Line int
// Raw is the (repaired, see RepairLegacyNumbers) JSON line.
Raw []byte
}
Event is one record of an events-NNNN.jsonl segment.
Common fields are parsed, every field (including the common ones) is available as raw JSON in Fields.
func ParseEvent ¶
ParseEvent parses one JSON line.
Lines written by older tdesktop builds under a locale with a comma decimal separator (`"t":12,345678`) are repaired, see RepairLegacyNumbers.
func (*Event) Decode ¶
Decode decodes the field into v. Returns false if the field is missing or can't be decoded.
type Field ¶
Field is a key and a JSON-encodable value of an event.
Use ID, SignedID and TypeID for 64-bit identifiers and constructor ids, so they are encoded the same way tdesktop writes them.
func StructFields ¶
StructFields converts a struct with json tags (e.g. Msg) into ordered fields, keeping the field order of the encoded JSON object.
type Header ¶
type Header struct {
V int `json:"v"`
Seg int `json:"seg"`
Run string `json:"run,omitempty"`
App string `json:"app,omitempty"`
Layer int `json:"layer,omitempty"`
APISHA256 string `json:"api_sha256,omitempty"`
MTProtoSHA256 string `json:"mtproto_sha256,omitempty"`
WallUTC string `json:"wall_utc,omitempty"`
MonoT float64 `json:"mono_t,omitempty"`
PID int64 `json:"pid,omitempty"`
Flags []string `json:"flags,omitempty"`
// UI is set when UI tracing (P14) is on.
UI *HeaderUI `json:"ui,omitempty"`
}
Header is the "hdr" record that starts every events segment.
type HeaderUI ¶
type HeaderUI struct {
V int `json:"v"`
// IDs is "hmac64" (ids written as "h:<16 hex>") or "raw" (decimal
// strings, with RawIDs).
IDs string `json:"ids,omitempty"`
RawIDs bool `json:"raw_ids"`
Verbose bool `json:"verbose,omitempty"`
}
HeaderUI is the "ui" object of "hdr": {"v":1,"ids":"hmac64","raw_ids":false}.
type ID ¶
type ID uint64
ID is a 64-bit identifier, encoded as a decimal JSON string.
Decoding also accepts JSON numbers.
func (ID) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*ID) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type IDKey ¶
type IDKey []byte
IDKey is the per-run key of hashed UI ids: HMAC-SHA256 over the decimal id, truncated to 64 bits.
func UIKey ¶
UIKey reads the ui.key of a run directory: 64 lowercase hex chars and a newline, as tdesktop writes it (32 raw bytes are also accepted). The error wraps os.ErrNotExist if there is no key.
type Inner ¶
type Inner struct {
MsgID ID `json:"msg_id"`
SeqNo int64 `json:"seqno"`
Len int64 `json:"len"`
Type TypeID `json:"type"`
Method TypeID `json:"method,omitempty"`
ReqMsgID ID `json:"req_msg_id,omitempty"`
Result TypeID `json:"result,omitempty"`
}
Inner describes one message of a msg_container, as summarized by the writer.
type Msg ¶
type Msg struct {
Dir string `json:"dir"`
SDC int `json:"sdc"`
DC int `json:"dc"`
Shift int `json:"shift"`
PDC int `json:"pdc"`
Conn string `json:"conn,omitempty"`
Key ID `json:"key"`
KeyHex string `json:"key_hex,omitempty"`
KeyKind string `json:"key_kind,omitempty"`
Session SignedID `json:"session"`
Salt ID `json:"salt"`
MsgID ID `json:"msg_id"`
SeqNo int64 `json:"seqno"`
Len int64 `json:"len"`
Truncated bool `json:"truncated,omitempty"`
Type TypeID `json:"type"`
Method TypeID `json:"method,omitempty"`
Wrap []TypeID `json:"wrap,omitempty"`
ReqMsgID ID `json:"req_msg_id,omitempty"`
Result TypeID `json:"result,omitempty"`
Inner []Inner `json:"inner,omitempty"`
Redacted []string `json:"redacted,omitempty"`
PII bool `json:"pii,omitempty"`
Partial bool `json:"partial,omitempty"`
}
Msg is the parsed form of a "msg" event.
type Run ¶
type Run struct {
// Dir is the run directory.
Dir string
// Segments are the indexes of present segments, sorted. Segments may
// have gaps: the writer deletes the oldest segments except 0 when the
// total size cap is reached.
Segments []int
// Header is the "hdr" record of the first present segment.
Header *Header
// contains filtered or unexported fields
}
Run is a trace run directory opened for reading.
type SignedID ¶
type SignedID int64
SignedID is a signed 64-bit identifier (session ids), encoded as a decimal JSON string.
func (SignedID) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*SignedID) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type TypeID ¶
type TypeID uint32
TypeID is a TL constructor id, encoded as "0x%08x".
func ParseTypeID ¶
ParseTypeID parses "0x1cb5c415" (or a decimal number) into TypeID.
func (TypeID) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*TypeID) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type UIStamp ¶
type UIStamp struct {
// A is the action id.
A uint32 `json:"a"`
// Rel is UIRelSync, UIRelDeferred or UIRelAfter.
Rel string `json:"rel"`
// DT is the time since the action scope closed (since it opened, for
// nested event loops), in integer milliseconds. It is written for
// deferred and after stamps.
DT float64 `json:"dt"`
}
UIStamp is the "ui" field of events emitted on the main thread during or shortly after an input action: {"a":41,"rel":"deferred","dt":0}.
func (UIStamp) MarshalJSON ¶
MarshalJSON implements json.Marshaler: "dt" is omitted for sync stamps.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes a trace run directory in the same format as the tdesktop tracer. It is safe for concurrent use.
func NewWriter ¶
func NewWriter(dir string, opt WriterOptions) (*Writer, error)
NewWriter creates dir (mode 0700) and opens segment 0.
func (*Writer) Write ¶
Write appends an event and returns its seq. If blob is not nil, it is written to the blobs segment and referenced as "blob" (and inlined as "b64" if it is small enough).
func (*Writer) WriteDropped ¶
WriteDropped appends a "dropped" record, as the tdesktop writer does when its queue overflows.
type WriterOptions ¶
type WriterOptions struct {
// Header are the fields of the "hdr" record (after "seq", "ev", "v" and
// "seg"), e.g. run, app, layer.
Header []Field
// SegmentEventsBytes rotates segments when the events file exceeds it.
// Default is 64 MiB.
SegmentEventsBytes int64
// SegmentBlobsBytes rotates segments when the blobs file exceeds it.
// Default is 256 MiB.
SegmentBlobsBytes int64
// TotalBytes caps the total size; the oldest segments except segment 0
// are deleted. Default is 2 GiB.
TotalBytes int64
// InlineMax also inlines bodies up to this size as "b64". Zero disables.
InlineMax int
// Now returns the event time in seconds. Default is the monotonic time
// since NewWriter.
Now func() float64
}
WriterOptions configures Writer.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package decode turns "msg" events of a wire trace into logical messages: containers are split, wrappers (invokeWithLayer, initConnection, invokeAfterMsg, ...) are unwrapped, rpc_result is joined with its request (method, latency, request id and caller).
|
Package decode turns "msg" events of a wire trace into logical messages: containers are split, wrappers (invokeWithLayer, initConnection, invokeAfterMsg, ...) are unwrapped, rpc_result is joined with its request (method, latency, request id and caller). |
|
Package tljson is a schema-driven generic TL decoder: it decodes binary TL into ordered JSON objects using api.tl and mtproto.tl parsed with github.com/gotd/tl, so it works for any layer without generated code.
|
Package tljson is a schema-driven generic TL decoder: it decodes binary TL into ordered JSON objects using api.tl and mtproto.tl parsed with github.com/gotd/tl, so it works for any layer without generated code. |
|
Package uiattr attributes requests of a wire trace to UI actions (P14): clicks, keys, scrolls, typing, shortcuts and menu items of the traced tdesktop build, and synthetic actions of the app.
|
Package uiattr attributes requests of a wire trace to UI actions (P14): clicks, keys, scrolls, typing, shortcuts and menu items of the traced tdesktop build, and synthetic actions of the app. |
|
Package wiretracetest builds synthetic wire traces for tests: TL bodies are encoded with gotd and written with wiretrace.Writer, with the same summary fields the tdesktop tracer writes.
|
Package wiretracetest builds synthetic wire traces for tests: TL bodies are encoded with gotd and written with wiretrace.Writer, with the same summary fields the tdesktop tracer writes. |