wiretrace

package
v0.164.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package wiretrace reads and writes wire traces: the plaintext of every encrypted MTProto message plus request, connection and storage lifecycle events, as written by the traced Telegram Desktop build (gotd/tdesktop fork, branch "trace").

Format

A run directory holds segments:

  • events-NNNN.jsonl: one JSON object per event. Every segment starts with a "hdr" record (seq 0) with "v", "seg", "run", "app", "layer", "api_sha256", "mtproto_sha256", "wall_utc", "mono_t", "pid" and "flags". Other records have "seq", "t" (monotonic seconds), "th" (thread) and "ev" (kind), plus kind-specific fields. 64-bit ids are decimal strings, session ids are signed, constructor ids are "0x%08x". Events emitted while a response or an update was handled carry "cause":{"req","outer","type"}. With UI tracing, "hdr" has "ui":{"v","ids","raw_ids"}, ui.* and app.* records describe user actions and view state, and main-thread events emitted during or shortly after an action carry "ui":{"a","rel","dt"} (UIStamp, see package wiretrace/uiattr).
  • blobs-NNNN.bin: "TDWT", u16 version 1, then records of u64 seq, u32 len, u32 crc32c and the bytes. Events reference them with "blob":{"seg","off","len","crc32c","sha256"}, where off points at the bytes after the record header. Small bodies may also be inlined as "b64".

Segments rotate at 64 MiB of events or 256 MiB of blobs. When the total size cap is reached the oldest segments are deleted, except segment 0. If the writer queue overflows, a "dropped" record with "count" is written.

"msg" events carry the body of one encrypted message (after decryption, before encryption) with the header fields (sdc, session, salt, msg_id, seqno, len) and a summary (type, method, wrap, inner, req_msg_id, result). Bodies of sensitive requests and responses are redacted in place, keeping lengths: [id]RedactedMarker[id][zeros...], listed in "redacted".

Older tdesktop trace builds wrote numbers with the process locale decimal separator ("t":12,345678); ParseEvent repairs such lines.

Usage

run, err := wiretrace.Open(dir)
...
err = run.Events(func(ev *wiretrace.Event) error {
	body, err := run.Blob(ev) // verified against crc32c and sha256
	...
}, nil)

Package wiretrace/tljson decodes TL bodies with the schema of the traced build, wiretrace/decode turns "msg" events into logical messages, and cmd/wiretrace is the command line tool (decode, timeline, stats, diff). Writer produces the same format, for tests and other emitters.

Index

Constants

View Source
const (
	// Version is the format version written in "hdr" records and in the blob
	// file header.
	Version = 1

	// BlobMagic starts every blobs-NNNN.bin file, followed by u16 Version.
	BlobMagic = "TDWT"
	// BlobFileHeaderSize is the size of the blob file header.
	BlobFileHeaderSize = 6
	// BlobRecordHeaderSize is the size of a blob record header:
	// u64 seq, u32 len, u32 crc32c (little-endian).
	BlobRecordHeaderSize = 16

	// RedactedMarker is written right after the constructor id of a redacted
	// TL object: [id][RedactedMarker][id][zeros...]. Lengths are kept.
	RedactedMarker uint32 = 0x7EDAC7ED
)

Format constants of a trace run directory.

View Source
const (
	KindHeader  = "hdr"
	KindDropped = "dropped"
	KindMsg     = "msg"

	KindReqNew     = "req.new"
	KindReqMsg     = "req.msg"
	KindReqResend  = "req.resend"
	KindReqDone    = "req.done"
	KindReqFail    = "req.fail"
	KindReqCancel  = "req.cancel"
	KindReqCaller  = "req.caller"
	KindReqMigrate = "req.migrate"

	KindState         = "state"
	KindSessionReset  = "session.reset"
	KindSessionNewID  = "session.new_id"
	KindSaltChange    = "salt.change"
	KindKeySet        = "key.set"
	KindKeyCreated    = "key.created"
	KindKeyDestroy    = "key.destroy_temp"
	KindConnStart     = "conn.start"
	KindConnUp        = "conn.up"
	KindConnDown      = "conn.down"
	KindConnReason    = "conn.reason"
	KindConnRestart   = "conn.restart"
	KindConfigLoaded  = "config.loaded"
	KindStorageWrite  = "st.write"
	KindStorageRaw    = "st.write.raw"
	KindStorageEnc    = "st.encrypt"
	KindStorageCommit = "st.commit"
	KindStorageRemove = "st.remove"
	KindStorageRead   = "st.read"
	KindStorageDec    = "st.decrypt"
)

Event kinds written by the tdesktop tracer.

View Source
const (
	KindUIPress     = "ui.press"
	KindUIClick     = "ui.click"
	KindUIKey       = "ui.key"
	KindUIKeyEnd    = "ui.key.end"
	KindUIScroll    = "ui.scroll"
	KindUIScrollEnd = "ui.scroll.end"
	KindUIType      = "ui.type"
	KindUITypeEnd   = "ui.type.end"
	KindUIShortcut  = "ui.shortcut"
	KindUIMenu      = "ui.menu"
	KindUIMarker    = "ui.marker"

	KindUIChat       = "ui.chat"
	KindUISection    = "ui.section"
	KindUIBox        = "ui.box"
	KindUIMediaOpen  = "ui.media.open"
	KindUIMediaClose = "ui.media.close"
	KindUICall       = "ui.call"
	KindUIAccount    = "ui.account"
	KindUISession    = "ui.session"
	KindUISearchChat = "ui.search.chat"
	KindUIFolder     = "ui.folder"
	KindUINav        = "ui.nav"
	KindAppWindow    = "app.window"
	KindAppState     = "app.state"
)

UI event kinds (P14). Input actions carry "a", the action id; semantic and view-state events carry "ui" (the action they came from) or "a" with "src":"app" when they have no input parent.

View Source
const (
	// UIRelSync: emitted inside the input scope (exact).
	UIRelSync = "sync"
	// UIRelDeferred: emitted shortly (kDeferredMs, 150 ms) after the last
	// input scope closed, with no newer input in between.
	UIRelDeferred = "deferred"
	// UIRelAfter: emitted within kAfterMs (3 s) after the last input scope,
	// or inside a scope opened more than 500 ms ago (a nested event loop,
	// dt counts from the scope open).
	UIRelAfter = "after"
)

Relations of a UIStamp to its action.

View Source
const (
	DirOut = "out"
	DirIn  = "in"
)

Directions of "msg" events.

View Source
const (
	PeerUser    = "user"
	PeerChat    = "chat"
	PeerChannel = "channel"
)

Peer kinds of PeerValue.

View Source
const HashPrefix = "h:"

HashPrefix starts hashed UI ids: "h:<16 hex>".

View Source
const MaxLineSize = 64 << 20

MaxLineSize limits the size of one events line.

View Source
const UIKeyName = "ui.key"

UIKeyName is the file of the per-run HMAC key of UI ids in a run directory. It is never written into events.

Variables

View Source
var ErrBlobMismatch = errors.New("blob mismatch")

ErrBlobMismatch is returned when a blob record does not match its reference.

Functions

func BlobsName

func BlobsName(segment int) string

BlobsName returns the name of the blobs file of segment.

func CRC32C

func CRC32C(data []byte) uint32

CRC32C computes the crc32c (Castagnoli) checksum used by blob records.

func EventsName

func EventsName(segment int) string

EventsName returns the name of the events file of segment.

func PeerValue

func PeerValue(kind string, id int64) uint64

PeerValue returns the tdesktop PeerId.value of a bare id: the id with the type in bits 48+ (user 0, chat 1, channel 2).

func RepairLegacyNumbers

func RepairLegacyNumbers(line []byte) ([]byte, bool)

RepairLegacyNumbers rewrites numbers like `:12,345678` (comma decimal separator, written by tdesktop trace builds before the locale fix) into `:12.345678` when they are followed by `,`, `}` or `]`.

Valid JSON never has `:<digits>,<digits>` followed by one of those characters outside of strings, so the repair is applied only to lines that failed to parse. Returns false if nothing was changed.

func VerifyBlob

func VerifyBlob(seq uint64, ref *BlobRef, header, body []byte) error

VerifyBlob checks a blob record header and body against its reference.

Types

type BlobRef

type BlobRef struct {
	Seg    int    `json:"seg"`
	Off    int64  `json:"off"`
	Len    int    `json:"len"`
	CRC32C uint32 `json:"crc32c"`
	SHA256 string `json:"sha256,omitempty"`
}

BlobRef references a body in a blobs-NNNN.bin segment.

Off is the offset of the body itself, after the 16-byte record header.

type Cause

type Cause struct {
	// Req is the request id of the response, 0 for updates.
	Req int64 `json:"req"`
	// Outer is the outer msg_id of the response.
	Outer ID `json:"outer"`
	// Type is the constructor id of the response.
	Type TypeID `json:"type"`
}

Cause is the causal context of an event emitted while a response or an update was being handled.

type Event

type Event struct {
	Seq    uint64
	T      float64
	Thread string
	Kind   string
	// Blob references the body of the event, if any.
	Blob *BlobRef
	// Inline is the decoded "b64" field, if the writer inlined the body.
	Inline []byte
	// Cause is the causal context, if any.
	Cause *Cause
	// UI is the UI action stamp, if any. It is never set for "hdr" records
	// (their "ui" is HeaderUI).
	UI *UIStamp
	// Fields holds all fields of the record.
	Fields map[string]json.RawMessage

	// Segment is the index of the events segment the record was read from.
	Segment int
	// Line is the 1-based line number in that segment.
	Line int
	// Raw is the (repaired, see RepairLegacyNumbers) JSON line.
	Raw []byte
}

Event is one record of an events-NNNN.jsonl segment.

Common fields are parsed, every field (including the common ones) is available as raw JSON in Fields.

func ParseEvent

func ParseEvent(line []byte) (*Event, error)

ParseEvent parses one JSON line.

Lines written by older tdesktop builds under a locale with a comma decimal separator (`"t":12,345678`) are repaired, see RepairLegacyNumbers.

func (*Event) Bool

func (e *Event) Bool(key string) bool

Bool returns a boolean field, or false.

func (*Event) Decode

func (e *Event) Decode(key string, v any) bool

Decode decodes the field into v. Returns false if the field is missing or can't be decoded.

func (*Event) Float

func (e *Event) Float(key string) float64

Float returns a number field, or 0.

func (*Event) Has

func (e *Event) Has(key string) bool

Has reports whether the field is present.

func (*Event) Header

func (e *Event) Header() (*Header, error)

Header parses a "hdr" event.

func (*Event) ID

func (e *Event) ID(key string) uint64

ID returns an unsigned 64-bit id field (decimal string or number), or 0.

func (*Event) Int

func (e *Event) Int(key string) int64

Int returns an integer field (JSON number or decimal string), or 0.

func (*Event) Msg

func (e *Event) Msg() (*Msg, error)

Msg parses a "msg" event.

func (*Event) Str

func (e *Event) Str(key string) string

Str returns a string field, or "" if missing.

func (*Event) Type

func (e *Event) Type(key string) uint32

Type returns a constructor id field ("0x%08x"), or 0.

type Field

type Field struct {
	Key   string
	Value any
}

Field is a key and a JSON-encodable value of an event.

Use ID, SignedID and TypeID for 64-bit identifiers and constructor ids, so they are encoded the same way tdesktop writes them.

func F

func F(key string, value any) Field

F is a shorthand for Field{Key: key, Value: value}.

func StructFields

func StructFields(v any) ([]Field, error)

StructFields converts a struct with json tags (e.g. Msg) into ordered fields, keeping the field order of the encoded JSON object.

type Header struct {
	V             int      `json:"v"`
	Seg           int      `json:"seg"`
	Run           string   `json:"run,omitempty"`
	App           string   `json:"app,omitempty"`
	Layer         int      `json:"layer,omitempty"`
	APISHA256     string   `json:"api_sha256,omitempty"`
	MTProtoSHA256 string   `json:"mtproto_sha256,omitempty"`
	WallUTC       string   `json:"wall_utc,omitempty"`
	MonoT         float64  `json:"mono_t,omitempty"`
	PID           int64    `json:"pid,omitempty"`
	Flags         []string `json:"flags,omitempty"`
	// UI is set when UI tracing (P14) is on.
	UI *HeaderUI `json:"ui,omitempty"`
}

Header is the "hdr" record that starts every events segment.

type HeaderUI

type HeaderUI struct {
	V int `json:"v"`
	// IDs is "hmac64" (ids written as "h:<16 hex>") or "raw" (decimal
	// strings, with RawIDs).
	IDs     string `json:"ids,omitempty"`
	RawIDs  bool   `json:"raw_ids"`
	Verbose bool   `json:"verbose,omitempty"`
}

HeaderUI is the "ui" object of "hdr": {"v":1,"ids":"hmac64","raw_ids":false}.

type ID

type ID uint64

ID is a 64-bit identifier, encoded as a decimal JSON string.

Decoding also accepts JSON numbers.

func (ID) MarshalJSON

func (i ID) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*ID) UnmarshalJSON

func (i *ID) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler.

type IDKey

type IDKey []byte

IDKey is the per-run key of hashed UI ids: HMAC-SHA256 over the decimal id, truncated to 64 bits.

func UIKey

func UIKey(dir string) (IDKey, error)

UIKey reads the ui.key of a run directory: 64 lowercase hex chars and a newline, as tdesktop writes it (32 raw bytes are also accepted). The error wraps os.ErrNotExist if there is no key.

func (IDKey) Hash

func (k IDKey) Hash(value uint64) string

Hash returns the hashed form "h:<16 hex>" of a decimal id value (for peers, the tdesktop PeerId.value, see PeerValue).

func (IDKey) Verify

func (k IDKey) Verify(h string, value uint64) bool

Verify reports whether h is the hashed form of value.

type Inner

type Inner struct {
	MsgID    ID     `json:"msg_id"`
	SeqNo    int64  `json:"seqno"`
	Len      int64  `json:"len"`
	Type     TypeID `json:"type"`
	Method   TypeID `json:"method,omitempty"`
	ReqMsgID ID     `json:"req_msg_id,omitempty"`
	Result   TypeID `json:"result,omitempty"`
}

Inner describes one message of a msg_container, as summarized by the writer.

type Msg

type Msg struct {
	Dir       string   `json:"dir"`
	SDC       int      `json:"sdc"`
	DC        int      `json:"dc"`
	Shift     int      `json:"shift"`
	PDC       int      `json:"pdc"`
	Conn      string   `json:"conn,omitempty"`
	Key       ID       `json:"key"`
	KeyHex    string   `json:"key_hex,omitempty"`
	KeyKind   string   `json:"key_kind,omitempty"`
	Session   SignedID `json:"session"`
	Salt      ID       `json:"salt"`
	MsgID     ID       `json:"msg_id"`
	SeqNo     int64    `json:"seqno"`
	Len       int64    `json:"len"`
	Truncated bool     `json:"truncated,omitempty"`
	Type      TypeID   `json:"type"`
	Method    TypeID   `json:"method,omitempty"`
	Wrap      []TypeID `json:"wrap,omitempty"`
	ReqMsgID  ID       `json:"req_msg_id,omitempty"`
	Result    TypeID   `json:"result,omitempty"`
	Inner     []Inner  `json:"inner,omitempty"`
	Redacted  []string `json:"redacted,omitempty"`
	PII       bool     `json:"pii,omitempty"`
	Partial   bool     `json:"partial,omitempty"`
}

Msg is the parsed form of a "msg" event.

type Run

type Run struct {
	// Dir is the run directory.
	Dir string
	// Segments are the indexes of present segments, sorted. Segments may
	// have gaps: the writer deletes the oldest segments except 0 when the
	// total size cap is reached.
	Segments []int
	// Header is the "hdr" record of the first present segment.
	Header *Header
	// contains filtered or unexported fields
}

Run is a trace run directory opened for reading.

func Open

func Open(dir string) (*Run, error)

Open opens a run directory.

func (*Run) All

func (r *Run) All() ([]*Event, error)

All reads all records of the run.

func (*Run) Blob

func (r *Run) Blob(ev *Event) ([]byte, error)

Blob returns the body of the event, verifying the record header (seq, len, crc32c) and the sha256 of the reference.

If the blob segment is missing but the body was inlined ("b64"), the inline body is returned. Returns nil, nil for events without a body.

func (*Run) Close

func (r *Run) Close() error

Close releases opened blob files.

func (*Run) Events

func (r *Run) Events(fn func(ev *Event) error, onBad func(segment, line int, err error)) error

Events calls fn for every record of every segment in file order, including "hdr" records. Lines that can't be parsed are reported via onBad (if not nil) and skipped.

type SignedID

type SignedID int64

SignedID is a signed 64-bit identifier (session ids), encoded as a decimal JSON string.

func (SignedID) MarshalJSON

func (i SignedID) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*SignedID) UnmarshalJSON

func (i *SignedID) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler.

type TypeID

type TypeID uint32

TypeID is a TL constructor id, encoded as "0x%08x".

func ParseTypeID

func ParseTypeID(s string) (TypeID, error)

ParseTypeID parses "0x1cb5c415" (or a decimal number) into TypeID.

func (TypeID) MarshalJSON

func (t TypeID) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (TypeID) String

func (t TypeID) String() string

String implements fmt.Stringer.

func (*TypeID) UnmarshalJSON

func (t *TypeID) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler.

type UIStamp

type UIStamp struct {
	// A is the action id.
	A uint32 `json:"a"`
	// Rel is UIRelSync, UIRelDeferred or UIRelAfter.
	Rel string `json:"rel"`
	// DT is the time since the action scope closed (since it opened, for
	// nested event loops), in integer milliseconds. It is written for
	// deferred and after stamps.
	DT float64 `json:"dt"`
}

UIStamp is the "ui" field of events emitted on the main thread during or shortly after an input action: {"a":41,"rel":"deferred","dt":0}.

func (UIStamp) MarshalJSON

func (s UIStamp) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler: "dt" is omitted for sync stamps.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes a trace run directory in the same format as the tdesktop tracer. It is safe for concurrent use.

func NewWriter

func NewWriter(dir string, opt WriterOptions) (*Writer, error)

NewWriter creates dir (mode 0700) and opens segment 0.

func (*Writer) Close

func (w *Writer) Close() error

Close flushes and closes the current segment.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush flushes buffered data to the files.

func (*Writer) Write

func (w *Writer) Write(thread, kind string, fields []Field, blob []byte) (uint64, error)

Write appends an event and returns its seq. If blob is not nil, it is written to the blobs segment and referenced as "blob" (and inlined as "b64" if it is small enough).

func (*Writer) WriteDropped

func (w *Writer) WriteDropped(count uint64) (uint64, error)

WriteDropped appends a "dropped" record, as the tdesktop writer does when its queue overflows.

func (*Writer) WriteMsg

func (w *Writer) WriteMsg(thread string, m *Msg, body []byte) (uint64, error)

WriteMsg appends a "msg" event with the plaintext body.

type WriterOptions

type WriterOptions struct {
	// Header are the fields of the "hdr" record (after "seq", "ev", "v" and
	// "seg"), e.g. run, app, layer.
	Header []Field
	// SegmentEventsBytes rotates segments when the events file exceeds it.
	// Default is 64 MiB.
	SegmentEventsBytes int64
	// SegmentBlobsBytes rotates segments when the blobs file exceeds it.
	// Default is 256 MiB.
	SegmentBlobsBytes int64
	// TotalBytes caps the total size; the oldest segments except segment 0
	// are deleted. Default is 2 GiB.
	TotalBytes int64
	// InlineMax also inlines bodies up to this size as "b64". Zero disables.
	InlineMax int
	// Now returns the event time in seconds. Default is the monotonic time
	// since NewWriter.
	Now func() float64
}

WriterOptions configures Writer.

Directories

Path Synopsis
Package decode turns "msg" events of a wire trace into logical messages: containers are split, wrappers (invokeWithLayer, initConnection, invokeAfterMsg, ...) are unwrapped, rpc_result is joined with its request (method, latency, request id and caller).
Package decode turns "msg" events of a wire trace into logical messages: containers are split, wrappers (invokeWithLayer, initConnection, invokeAfterMsg, ...) are unwrapped, rpc_result is joined with its request (method, latency, request id and caller).
Package tljson is a schema-driven generic TL decoder: it decodes binary TL into ordered JSON objects using api.tl and mtproto.tl parsed with github.com/gotd/tl, so it works for any layer without generated code.
Package tljson is a schema-driven generic TL decoder: it decodes binary TL into ordered JSON objects using api.tl and mtproto.tl parsed with github.com/gotd/tl, so it works for any layer without generated code.
Package uiattr attributes requests of a wire trace to UI actions (P14): clicks, keys, scrolls, typing, shortcuts and menu items of the traced tdesktop build, and synthetic actions of the app.
Package uiattr attributes requests of a wire trace to UI actions (P14): clicks, keys, scrolls, typing, shortcuts and menu items of the traced tdesktop build, and synthetic actions of the app.
Package wiretracetest builds synthetic wire traces for tests: TL bodies are encoded with gotd and written with wiretrace.Writer, with the same summary fields the tdesktop tracer writes.
Package wiretracetest builds synthetic wire traces for tests: TL bodies are encoded with gotd and written with wiretrace.Writer, with the same summary fields the tdesktop tracer writes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL