Documentation
¶
Overview ¶
Package updates provides a Telegram's internalState synchronization manager.
It guarantees that all internalState-sensitive updates will be performed in correct order.
Limitations:
Manager cannot verify stateless types of updates (tg.UpdatesClass without Seq, or tg.UpdateClass without Pts or Qts).
Due to the fact that updates.getDifference and updates.getChannelDifference do not return event sequences, manager cannot guarantee the correctness of these operations. We rely on the server here.
Manager cannot fully recover the channel gap if there is a ChannelDifferenceTooLong error. The gap range and the latest messages returned by the server are reported via Config.OnChannelGap (not the handler, as they may have been delivered already), so missed messages can be fetched manually. See: https://core.telegram.org/constructor/updates.channelDifferenceTooLong
Failed differences ¶
A failed updates.getDifference or updates.getChannelDifference is retried with backoff (1 s doubling up to 60 s, then every 60–80 s), like TDLib and Telegram Desktop; FLOOD_WAIT is waited out. A lost authorization is not retried. PERSISTENT_TIMESTAMP_INVALID and PERSISTENT_TIMESTAMP_EMPTY on the common sequence reset the state from updates.getState and call Config.OnTooLong, as the updates in between are lost. CHANNEL_INVALID stops tracking the channel, like CHANNEL_PRIVATE. Config.Clock drives the retries.
Observability ¶
Events are logged via Config.Logger: debug logs for every difference request, gap, tracked channel and update received before Run, warnings for abnormal ones (e.g. a channel gap whose pts did not advance).
With Config.MeterProvider, the manager records these OpenTelemetry metrics (meter "github.com/gotd/td/telegram/updates"). Attributes never carry channel or user IDs.
- gotd.updates.difference.requests (counter): getDifference (kind=common) and getChannelDifference (kind=channel) calls by result: empty, slice, full, too_long, error, inaccessible.
- gotd.updates.difference.duration (histogram, s): duration of those calls, same attributes.
- gotd.updates.difference.triggers (counter): why the difference was requested, by kind and reason (e.g. startup, pts-gap-timeout, idle-timeout, channel-too-long, and retry or channel-retry for a retry of a failed difference).
- gotd.updates.gaps (counter): sequence gaps by kind, seq (pts, qts, seq) and action: detected, resolved_by_waiting, resolved_by_pending. Gaps not resolved in time trigger a difference with a *-gap-timeout reason.
- gotd.updates.channel.too_long (counter): channelDifferenceTooLong, i.e. Config.OnChannelGap calls, by dialog: ok, stale (pts did not advance), invalid (no dialog pts).
- gotd.updates.channel.gap.size (histogram, {pts}): RemotePts - LocalPts of those gaps.
- gotd.updates.channel.gap.messages (histogram, {message}): length of ChannelGap.Messages.
- gotd.updates.before_run (counter, {batch}): updates received before Run by action: passed, buffered, dropped, replayed, passed_raw (replayed to the handler as-is because the state was fetched from the server).
- gotd.updates.channel.track (counter): channel tracking by source (api for Manager.TrackChannel, stored, update, too_long) and result (started, already, pending, stopped, canceled, error).
- gotd.updates.channels.tracked (up-down counter): channels tracked by running managers.
- gotd.updates.channel.inaccessible (counter): channels removed from tracking after CHANNEL_PRIVATE or CHANNEL_INVALID.
- gotd.updates.stopped (counter): calls that returned ErrStopped, by op: handle, handle_affected, track_channel.
- gotd.updates.dispatched (counter, {update}): updates passed to the handler, by source: live or difference.
Index ¶
- Variables
- type API
- type AuthOptions
- type BeforeRunMode
- type ChannelAccessHasher
- type ChannelGap
- type Config
- type FileStorage
- func (s *FileStorage) ForEachChannels(ctx context.Context, userID int64, ...) error
- func (s *FileStorage) GetChannelAccessHash(_ context.Context, userID, channelID int64) (int64, bool, error)
- func (s *FileStorage) GetChannelPts(_ context.Context, userID, channelID int64) (int, bool, error)
- func (s *FileStorage) GetState(_ context.Context, userID int64) (State, bool, error)
- func (s *FileStorage) GetUserAccessHash(_ context.Context, userID, targetUserID int64) (int64, bool, error)
- func (s *FileStorage) SetChannelAccessHash(_ context.Context, userID, channelID, accessHash int64) error
- func (s *FileStorage) SetChannelPts(_ context.Context, userID, channelID int64, pts int) error
- func (s *FileStorage) SetDate(_ context.Context, userID int64, date int) error
- func (s *FileStorage) SetDateSeq(_ context.Context, userID int64, date, seq int) error
- func (s *FileStorage) SetPts(_ context.Context, userID int64, pts int) error
- func (s *FileStorage) SetQts(_ context.Context, userID int64, qts int) error
- func (s *FileStorage) SetSeq(_ context.Context, userID int64, seq int) error
- func (s *FileStorage) SetState(_ context.Context, userID int64, state State) error
- func (s *FileStorage) SetUserAccessHash(_ context.Context, userID, targetUserID, accessHash int64) error
- type Manager
- func (m *Manager) Handle(ctx context.Context, u tg.UpdatesClass) error
- func (m *Manager) HandleAffected(ctx context.Context, channelID int64, pts, ptsCount int) error
- func (m *Manager) Reset()
- func (m *Manager) Run(ctx context.Context, api API, userID int64, opt AuthOptions) error
- func (m *Manager) TrackChannel(ctx context.Context, channelID, accessHash int64, pts int) error
- type State
- type StateStorage
- type UserAccessHasher
Constants ¶
This section is empty.
Variables ¶
var ErrStopped = errors.New("updates manager stopped")
ErrStopped is returned by Handle, HandleAffected and TrackChannel after Run stopped, until Reset.
Functions ¶
This section is empty.
Types ¶
type API ¶ added in v0.80.0
type API interface {
UpdatesGetState(ctx context.Context) (*tg.UpdatesState, error)
UpdatesGetDifference(ctx context.Context, request *tg.UpdatesGetDifferenceRequest) (tg.UpdatesDifferenceClass, error)
UpdatesGetChannelDifference(ctx context.Context, request *tg.UpdatesGetChannelDifferenceRequest) (tg.UpdatesChannelDifferenceClass, error)
}
API is the interface which contains Telegram RPC methods used by manager for internalState synchronization.
type AuthOptions ¶ added in v0.80.0
type BeforeRunMode ¶
type BeforeRunMode int
BeforeRunMode is what Manager does with updates received before Run.
const ( // BeforeRunPass passes updates received before Run to Handler as-is, // without order verification, gap recovery or short updates // transformation. Default. BeforeRunPass BeforeRunMode = iota // BeforeRunBuffer keeps updates received before Run, at most // Config.BeforeRunBufferSize batches, and pushes them through gap recovery // once Run starts. Updates over the limit are dropped. Updates received // while they are replayed are buffered too, so they are handled in order. // // If Run fetches the state from the server (AuthOptions.Forget, or no // stored state), the buffered updates are already covered by it, so they // are passed to Handler as-is, like with BeforeRunPass. // // Use it when handlers are ready only after authorization, e.g. // UpdateDispatcher handlers registered in Run's OnStart. BeforeRunBuffer // BeforeRunDrop drops updates received before Run. BeforeRunDrop )
type ChannelAccessHasher ¶ added in v0.51.0
type ChannelAccessHasher interface {
SetChannelAccessHash(ctx context.Context, userID, channelID, accessHash int64) error
GetChannelAccessHash(ctx context.Context, userID, channelID int64) (accessHash int64, found bool, err error)
}
ChannelAccessHasher stores users channel access hashes.
type ChannelGap ¶
type ChannelGap struct {
// ChannelID is the channel ID.
ChannelID int64
// AccessHash is the channel access hash.
AccessHash int64
// LocalPts is the local pts before the gap.
LocalPts int
// RemotePts is the new pts after the gap (pts of the channel dialog).
RemotePts int
// TopMessage is the ID of the latest channel message (from the
// channel dialog), zero if unknown.
TopMessage int
// Messages are the latest channel messages returned with the gap. They
// may include messages delivered to Handler before the gap, so they are
// not passed to Handler.
Messages []tg.MessageClass
// Users and Chats are entities of Messages.
Users []tg.UserClass
Chats []tg.ChatClass
}
ChannelGap describes a channel gap that the manager cannot recover (updates.channelDifferenceTooLong).
type Config ¶
type Config struct {
// Handler where updates will be passed.
Handler telegram.UpdateHandler
// StopOnApplyError makes Run return on Handler or StateStorage errors
// while applying updates, instead of logging them and continuing
// (default).
//
// When enabled, a failed Handler call is not followed by the checkpoint
// write of its apply path, so the pts, qts, seq or channel pts is not
// advanced in memory or in Storage; a failed StateStorage write does not
// advance the corresponding in-memory state; a difference slice is not
// continued. Errors of RPC calls keep their usual handling.
//
// This is an error policy, not a transaction: completed steps and
// concurrent work (e.g. other channels) are not rolled back, a Storage
// may commit before returning an error, and a handler may have partially
// applied the failed batch. Handlers must be idempotent, as updates
// after the last successful checkpoint are delivered again. After
// resolving the failure, restart with a new Manager and Forget=false to
// resume from the stored state.
//
// Updates outside any sequence that a channel difference forwards to the
// common state are handed off before the channel pts is stored, so a
// failure to handle them does not hold that pts back.
StopOnApplyError bool
// Callback called if manager cannot
// recover channel gap (optional).
//
// See also OnChannelGap, which provides the gap range.
OnChannelTooLong func(channelID int64)
// Callback called if manager cannot recover channel gap, i.e. on
// updates.channelDifferenceTooLong, with the gap range (optional).
//
// The latest channel messages returned by the server with
// channelDifferenceTooLong are passed in gap.Messages, not to Handler:
// they may include messages delivered before the gap. Messages between
// the last known message and gap.Messages should be fetched manually,
// e.g. via messages.getHistory.
//
// Called after OnChannelTooLong.
OnChannelGap func(gap ChannelGap)
// Callback called when the manager loses access to a channel, detected via
// CHANNEL_PRIVATE or CHANNEL_INVALID on updates.getChannelDifference (e.g. the account was
// kicked/banned or the channel was deleted). The channel is removed from
// the update manager after this call (optional).
OnChannelInaccessible func(channelID int64)
// Callback called if manager cannot recover
// common state gap, i.e. on updates.differenceTooLong (optional).
OnTooLong func()
// Callback called when the manager fails to load the locally stored user
// state during Run with forget=false (no state found), so the state is
// fetched from the server and a full resynchronization is performed.
//
// Useful to detect that updates missed while offline may need to be fetched
// manually, e.g. via messages.getHistory (optional).
OnLoadUserStateFailed func()
// Callback called when the manager fails to load the locally stored state of
// a channel during Run with forget=false (its access hash is missing), so
// the channel is skipped.
//
// Useful to detect that the channel may need to be resynchronized manually,
// e.g. by calling Manager.TrackChannel with its access hash, which resumes
// from the stored pts (optional).
OnLoadChannelStateFailed func(channelID int64)
// State storage.
// In-mem used if not provided.
Storage StateStorage
// Channel access hash storage. In-mem used if not provided.
//
// The manager needs the access hash of a channel to track it. Hashes are
// learned from the entities of updates and differences; for a channel
// the manager has not seen there yet (e.g. known only from dialogs), its
// first update costs an extra updates.getDifference to find the hash.
// Pass a hasher that also knows the peers the rest of the app has seen,
// such as *peers.Manager, or a persistent one (FileStorage, the contrib
// storages), to avoid that and to keep channels across restarts.
AccessHasher ChannelAccessHasher
// User access hash storage. If not provided, AccessHasher is used when
// it implements UserAccessHasher (e.g. *peers.Manager, FileStorage),
// otherwise an in-mem one.
UserAccessHasher UserAccessHasher
// Logger (optional).
Logger log.Logger
// TracerProvider (optional).
TracerProvider trace.TracerProvider
// MeterProvider for metrics of the manager (optional). No metrics by
// default. See the package documentation for the list.
MeterProvider metric.MeterProvider
// MaxChannelDifferenceConcurrency limits how many updates.getChannelDifference
// requests may be in flight at once across all tracked channels. Non-positive
// values (<= 0, default 0) mean unlimited: every channel recovers its gap as
// before. A positive value bounds the burst so an account in many active
// channels does not exceed Telegram's per-account method rate limit.
MaxChannelDifferenceConcurrency int
// BeforeRun is what to do with updates received before Run, e.g. while
// the client connects and authorizes. Default is BeforeRunPass.
//
// Dropped updates (BeforeRunDrop, or over BeforeRunBufferSize) of the
// common or a tracked channel sequence are recovered via getDifference
// once the next update reveals the gap.
BeforeRun BeforeRunMode
// BeforeRunBufferSize limits how many update batches are kept with
// BeforeRunBuffer. Non-positive means 100.
BeforeRunBufferSize int
// Clock is used for retries of failed differences. Defaults to
// clock.System.
Clock clock.Clock
}
Config of the manager.
type FileStorage ¶
type FileStorage struct {
// contains filtered or unexported fields
}
FileStorage is a StateStorage, ChannelAccessHasher and UserAccessHasher that keeps the update state of every user, the pts of their channels and the channel and user access hashes in one JSON file, so that a restarted client gets the updates it missed while stopped.
Every change rewrites the file atomically (a temporary file in the same directory, then rename), so the file is never left half-written, but a crash of the machine may lose the last changes and lead to updates being delivered again. It suits bots and small accounts; for accounts in many active channels use a database-backed storage, e.g. from gotd/contrib.
Use the same FileStorage as Config.Storage and Config.AccessHasher: channel state is restored on start only together with its access hash.
A FileStorage must not be shared by processes. It is safe for concurrent use.
func NewFileStorage ¶
func NewFileStorage(path string) *FileStorage
NewFileStorage returns a FileStorage that keeps the state in the file at path. The file is created on the first change, and its directory must exist.
func (*FileStorage) ForEachChannels ¶
func (s *FileStorage) ForEachChannels( ctx context.Context, userID int64, f func(ctx context.Context, channelID int64, pts int) error, ) error
ForEachChannels implements StateStorage.
func (*FileStorage) GetChannelAccessHash ¶
func (s *FileStorage) GetChannelAccessHash(_ context.Context, userID, channelID int64) (int64, bool, error)
GetChannelAccessHash implements ChannelAccessHasher.
func (*FileStorage) GetChannelPts ¶
GetChannelPts implements StateStorage.
func (*FileStorage) GetUserAccessHash ¶
func (s *FileStorage) GetUserAccessHash(_ context.Context, userID, targetUserID int64) (int64, bool, error)
GetUserAccessHash implements UserAccessHasher.
func (*FileStorage) SetChannelAccessHash ¶
func (s *FileStorage) SetChannelAccessHash(_ context.Context, userID, channelID, accessHash int64) error
SetChannelAccessHash implements ChannelAccessHasher.
func (*FileStorage) SetChannelPts ¶
SetChannelPts implements StateStorage.
func (*FileStorage) SetDateSeq ¶
SetDateSeq implements StateStorage.
func (*FileStorage) SetState ¶
SetState implements StateStorage. It forgets the pts of the user's channels, but keeps their access hashes.
func (*FileStorage) SetUserAccessHash ¶
func (s *FileStorage) SetUserAccessHash(_ context.Context, userID, targetUserID, accessHash int64) error
SetUserAccessHash implements UserAccessHasher.
type Manager ¶ added in v0.51.0
type Manager struct {
// contains filtered or unexported fields
}
Manager deals with gaps.
Important: Updates produced by this manager may contain negative Pts/Qts/Seq values in tg.UpdateClass/tg.UpdatesClass (does not affects to the tg.MessageClass).
This is because telegram server does not return these sequences for getDifference/getChannelDifference results. You SHOULD NOT use them in update handlers at all.
func (*Manager) Handle ¶ added in v0.51.0
Handle handles updates.
Important: If Run method not called, updates are handled according to Config.BeforeRun: by default, they are passed to the provided handler as-is without any order verification or short updates transformation.
func (*Manager) HandleAffected ¶ added in v0.157.0
HandleAffected applies the pts increment carried by a messages.affectedMessages or messages.affectedHistory RPC result, keeping the local pts in sync after a self-initiated read or delete (e.g. messages.readHistory, messages.deleteMessages).
Without this, such results silently advance the server pts while the local pts stays behind, so the next genuine update looks like a gap and is postponed until the next pts-changing update or a getDifference (see issue #1382).
channelID must be the channel the result belongs to, or 0 for the common (user/basic-group) sequence. The affected pts for a channel is applied only when that channel is already tracked. It is a no-op before Run is called.
Most callers should not call this directly: use the hook middleware in telegram/updates/hook, which wires it from RPC results automatically.
func (*Manager) Reset ¶ added in v0.80.0
func (m *Manager) Reset()
Reset notifies manager about user logout.
func (*Manager) Run ¶ added in v0.80.0
Run notifies manager about user authentication on the telegram server.
If forget is true, local internalState (if exist) will be overwritten with remote internalState.
func (*Manager) TrackChannel ¶
TrackChannel starts tracking updates of the channel, if it is not tracked yet.
The manager tracks only channels with a stored pts and access hash, and learns about other channels only from their next update, so messages posted before it are missed. Use TrackChannel to track channels joined while the client was offline, or channels reported via Config.OnLoadChannelStateFailed, e.g. with pts of the channel dialog from messages.getDialogs (Dialog.Pts).
If the channel has a stored pts, tracking starts from it, so missed updates are fetched via updates.getChannelDifference. Otherwise it starts from the given pts, which must be positive.
Can be called before Run or from AuthOptions.OnStart: such channels start being tracked when the manager starts, and errors are only logged.
type StateStorage ¶ added in v0.51.0
type StateStorage interface {
GetState(ctx context.Context, userID int64) (state State, found bool, err error)
SetState(ctx context.Context, userID int64, state State) error
SetPts(ctx context.Context, userID int64, pts int) error
SetQts(ctx context.Context, userID int64, qts int) error
SetDate(ctx context.Context, userID int64, date int) error
SetSeq(ctx context.Context, userID int64, seq int) error
SetDateSeq(ctx context.Context, userID int64, date, seq int) error
GetChannelPts(ctx context.Context, userID, channelID int64) (pts int, found bool, err error)
SetChannelPts(ctx context.Context, userID, channelID int64, pts int) error
ForEachChannels(ctx context.Context, userID int64, f func(ctx context.Context, channelID int64, pts int) error) error
}
StateStorage is the users internalState storage.
Note: SetPts, SetQts, SetDate, SetSeq, SetDateSeq should return error if user internalState does not exist.
type UserAccessHasher ¶ added in v0.150.0
type UserAccessHasher interface {
SetUserAccessHash(ctx context.Context, userID, targetUserID, accessHash int64) error
GetUserAccessHash(ctx context.Context, userID, targetUserID int64) (accessHash int64, found bool, err error)
}
UserAccessHasher stores user access hashes observed by the updates manager so it can decide whether an incoming update references a user it already knows. Mirrors ChannelAccessHasher. In-mem used if not provided to Config.
Source Files
¶
- access_hash_feeder.go
- apply_error.go
- before_run.go
- channel_diff_semaphore.go
- config.go
- conv_shorts.go
- diff_retry.go
- doc.go
- gap_buffer.go
- gap_check.go
- manager.go
- message_peer.go
- metrics.go
- min_backfill.go
- sequence_box.go
- short_peer.go
- sort_pts.go
- state.go
- state_apply.go
- state_channel.go
- state_difference.go
- storage.go
- storage_file.go
- storage_mem.go
- track_channel.go
- update.go
- utils.go