README
¶
Trickle Protocol
Trickle is a segmented publish-subscribe protocol that streams data in realtime, mainly over HTTP.
Breaking this down:
-
Data streams are called channels , Channels are content agnostic - the data can be video, audio, JSON, logs, a custom format, etc.
-
Segmented: Data for each channel is sent as discrete segments. For example, a video may use a segment for each GOP, or an API may use a segment for each JSON message, or a logging application may split segments on time intervals. The boundaries of each segment are application defined, but segments are preferably standalone such that a request for a single segment should return usable content on its own.
-
Publish-subscribe: Publishers push segments, and subscribers pull them.
-
Streaming: Content can be sent by publishers and received by subscribers in real-time as it is generated - data is trickled out.
-
HTTP: Trickle is tied to HTTP with GET / POST semantics, status codes, path based routing, metadata in headers, etc. However, other implementations are possible, such as a local, in-memory trickle client.
Protocol Specification
TODO: more details
Publishers POST to /channel-name/seq
Subscribers GET to /channel-name/seq
The channel-name is any valid HTTP path part.
The seq is an integer sequence number indicating the segment, and must increase sequentially without gaps.
As data is published to channel-name/seq,the server will relay the data to all subscribers for channel-name/seq.
Clients are responsible for maintaining their own position in the sequence.
Servers may opt to keep the last N segments for subscribers to catch up on.
Servers will 404 if a channel-name or a seq does not exist.
Clients may pre-connect the next segment in order to set up the resource and minimize connection set-up time.
Publishers should only actively send data to one seq at a time, although they may still pre-connect to seq + 1
Publishers do not have to push content immeditely after preconnecting, however the server should have some reasonable timeout to avoid excessive idle connections.
If a subscriber retrieves a segment mid-publish, the server should return all the content it has up until that point, and trickle down the rest as it receives it.
If a timeout has been hit without sending (or receiving) content, the publisher (or subscriber) can re-connect to the same seq. (TODO; also indicate timeout via signaling)
Servers may offer some grace with leading sequence numbers to avoid data races, eg allowing a GET for seq+1 if a publisher hasn't yet preconnected that number.
Publishers are responsible for segmenting content (if necessary) and subscribers are responsible for re-assembling content (if necessary)
Subscribers can initiate a subscribe with a seq of -1 to retrieve the most recent publish. With preconnects, the subscriber may be waiting for the next publish. For video this allows clients to eg, start streaming at the live edge of the next GOP.
Subscribers can retrieve the current seq with the Lp-Trickle-Seq metadata (HTTP header). This is useful in case -1 was used to initiate the subscription; the subscribing client can then pre-connect to Lp-Trickle-Seq + 1
Subscribers can initiate a subscribe with a seq of -N to get the Nth-from-last segment. (TODO)
The server should send subscribers Lp-Trickle-Size metadata to indicate the size of the content up until now. This allows clients to know where the live edge is, eg video implementations can decode-and-discard frames up until the edge to achieve immediate playback without waiting for the next segment. (TODO)
The server currently has a special changefeed channel named _changes which will send subscribers updates on streams that are added and removed. The changefeed is disabled by default.
Documentation
¶
Index ¶
- Constants
- Variables
- func GetLatest(resp *http.Response) int
- func GetSeq(resp *http.Response) int
- func IsEOS(resp *http.Response) bool
- type Changefeed
- type HTTPError
- type Segment
- type SegmentSubscriber
- type SequenceNonexistent
- type Server
- type Stream
- type TrickleData
- type TrickleLocalPublisher
- type TrickleLocalSubscriber
- type TricklePublisher
- type TrickleServerConfig
- type TrickleSubscriber
Constants ¶
const CHANGEFEED = "_changes"
Variables ¶
var EOS = errors.New("End of stream")
var FirstByteTimeout = errors.New("pending read timeout")
var StreamNotFoundErr = errors.New("stream not found")
Functions ¶
Types ¶
type Changefeed ¶
type SegmentSubscriber ¶
type SegmentSubscriber struct {
// contains filtered or unexported fields
}
type SequenceNonexistent ¶ added in v0.8.2
func (*SequenceNonexistent) Error ¶ added in v0.8.2
func (e *SequenceNonexistent) Error() string
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
func ConfigureServer ¶
func ConfigureServer(config TrickleServerConfig) *Server
type TrickleLocalPublisher ¶
type TrickleLocalPublisher struct {
// contains filtered or unexported fields
}
func NewLocalPublisher ¶
func NewLocalPublisher(sm *Server, channelName string, mimeType string) *TrickleLocalPublisher
func (*TrickleLocalPublisher) Close ¶
func (c *TrickleLocalPublisher) Close() error
func (*TrickleLocalPublisher) CreateChannel ¶
func (c *TrickleLocalPublisher) CreateChannel()
type TrickleLocalSubscriber ¶
type TrickleLocalSubscriber struct {
// contains filtered or unexported fields
}
func NewLocalSubscriber ¶
func NewLocalSubscriber(sm *Server, channelName string) *TrickleLocalSubscriber
func (*TrickleLocalSubscriber) Read ¶
func (c *TrickleLocalSubscriber) Read() (*TrickleData, error)
type TricklePublisher ¶
type TricklePublisher struct {
// contains filtered or unexported fields
}
TricklePublisher represents a trickle streaming client
func NewTricklePublisher ¶
func NewTricklePublisher(url string) (*TricklePublisher, error)
NewTricklePublisher creates a new trickle stream client
func (*TricklePublisher) Close ¶
func (c *TricklePublisher) Close() error
func (*TricklePublisher) Next ¶ added in v0.8.2
func (c *TricklePublisher) Next() (*pendingPost, error)
type TrickleServerConfig ¶
type TrickleServerConfig struct {
// Base HTTP path for the server
BasePath string
// HTTP mux to use
Mux *http.ServeMux
// Whether to enable the changefeed (default false)
Changefeed bool
// Whether to auto-create channels on first publish (default false)
Autocreate bool
// Amount of time a channel has no new segments before being swept (default 5 minutes)
IdleTimeout time.Duration
// How often to sweep for idle channels (default 1 minute)
SweepInterval time.Duration
}
type TrickleSubscriber ¶
type TrickleSubscriber struct {
// contains filtered or unexported fields
}
TrickleSubscriber represents a trickle streaming reader that always fetches from index -1
func NewTrickleSubscriber ¶
func NewTrickleSubscriber(url string) *TrickleSubscriber
NewTrickleSubscriber creates a new trickle stream reader for GET requests
func (*TrickleSubscriber) Read ¶
func (c *TrickleSubscriber) Read() (*http.Response, error)
Read retrieves data from the current segment and sets up the next segment concurrently. It returns the reader for the current segment's data.
func (*TrickleSubscriber) SetSeq ¶ added in v0.8.2
func (c *TrickleSubscriber) SetSeq(seq int)
Source Files