Documentation
¶
Overview ¶
Finding a stored message by what somebody knows about it.
The search a person actually needs is "here is a number off a phone call, show me every message about it". Before this that meant either a substring match over payloads, which is grep and slow, or a filter expression, which requires knowing that identity lives at PID-3 in HL7 v2 and somewhere else entirely in the other three formats on the same server.
Identity is extracted once when the message is recorded and indexed, so this is a lookup rather than a scan. The cost is paid on the recording path, once per message, instead of on every search.
Package msgstore persists the messages a channel handled.
This is the screen people leave open all day in an interface engine: what arrived, what happened to it, and the ability to look at one and send it again. Without it an engine can only answer "how many" and never "which one".
Retention is built in rather than added later. The most commonly reported operational failure of an interface engine is its message tables filling the disk, and a store with no pruning is that failure waiting for enough traffic.
Index ¶
- Constants
- Variables
- func DescribeHour(hour int) string
- func HourOfWeek(t time.Time) int
- func IdentityKinds() []identity.Kind
- func NormaliseIdentifiers(idents []string) []string
- type AttachmentInfo
- type AttachmentStats
- type Bucket
- type Delivery
- type DeliveryStatus
- type DestinationStats
- type ExpressionResult
- type ExpressionSearch
- type Flow
- type FlowChannel
- type HourStat
- type IdentityMatch
- type IdentityResult
- type IdentitySearch
- type LastDelivery
- type Message
- type MinuteCount
- type Outcome
- type Query
- type RecentFilter
- type RecentMessage
- type Recorder
- type Rhythm
- type Stats
- type Store
- func (s *Store) Attachment(ctx context.Context, digest attach.Digest) ([]byte, bool, error)
- func (s *Store) AttachmentLookup(ctx context.Context) attach.Lookup
- func (s *Store) AttachmentStats(ctx context.Context) (AttachmentStats, error)
- func (s *Store) Channels(ctx context.Context, tenantID string) ([]string, error)
- func (s *Store) CountPayloads(ctx context.Context, channel string) (int, error)
- func (s *Store) CountRecords(ctx context.Context, channel string) (int, error)
- func (s *Store) DestinationStats(ctx context.Context, tenantID string, since time.Time) ([]DestinationStats, error)
- func (s *Store) EraseMessages(ctx context.Context, ids []int64) (int64, error)
- func (s *Store) Flow(ctx context.Context, tenantID string, since time.Time, bucket time.Duration) (*Flow, error)
- func (s *Store) Get(ctx context.Context, tenantID string, id int64) (*Message, error)
- func (s *Store) IndexesIdentity() bool
- func (s *Store) LastDeliveries(ctx context.Context, tenantID string, since time.Time) ([]LastDelivery, error)
- func (s *Store) LearnRhythm(ctx context.Context, tenantID string, weeks int) (map[string]*Rhythm, error)
- func (s *Store) LinkAttachments(ctx context.Context, messageID int64, digests []attach.Digest) error
- func (s *Store) List(ctx context.Context, q Query) ([]Message, int, error)
- func (s *Store) MessageAttachments(ctx context.Context, messageID int64) ([]AttachmentInfo, error)
- func (s *Store) MinuteCounts(ctx context.Context, tenantID string, since time.Time) ([]MinuteCount, error)
- func (s *Store) Prune(ctx context.Context) (payloads, rows int64, err error)
- func (s *Store) PutAttachment(ctx context.Context, a attach.Attachment) error
- func (s *Store) Reader() *sql.DB
- func (s *Store) Recent(ctx context.Context, tenantID string, since time.Time, f RecentFilter) ([]RecentMessage, error)
- func (s *Store) RecentPayloads(ctx context.Context, channel string, limit int) ([][]byte, error)
- func (s *Store) Record(ctx context.Context, m *Message) (int64, error)
- func (s *Store) SearchByExpression(ctx context.Context, q ExpressionSearch) (*ExpressionResult, error)
- func (s *Store) SearchByIdentity(ctx context.Context, q IdentitySearch) (*IdentityResult, error)
- func (s *Store) SetHolds(held func(ctx context.Context) []string)
- func (s *Store) StartPruner(ctx context.Context, every time.Duration, ...) func()
- func (s *Store) Stats(ctx context.Context, tenantID string, since time.Time) (*Stats, error)
- func (s *Store) StoredBytes(ctx context.Context, tenantID string) (int64, error)
- func (s *Store) SubjectMessages(ctx context.Context, tenantID string, norms []string, limit int) ([]SubjectMessage, error)
- func (s *Store) SweepAttachments(ctx context.Context) (int64, error)
- func (s *Store) Throughput(ctx context.Context, tenantID string, since time.Time, bucket time.Duration, ...) ([]Bucket, error)
- type Strand
- type SubjectMessage
Constants ¶
const DefaultLimit = 100
DefaultLimit is the page size when none is given.
const HoursInWeek = 168
HoursInWeek is the number of buckets in a rhythm.
const MaxLimit = 1000
MaxLimit bounds a page, because an unbounded one asks the server to load the whole table into memory.
const MaxPayloadBatch = 10000
MaxPayloadBatch bounds one bulk read.
Ten thousand messages at a few kilobytes each is a few tens of megabytes held at once, which is acceptable for a deliberate action somebody waits on. It is also comfortably more traffic than anybody needs to be convinced by: a change that looks safe across ten thousand real messages and turns out not to be was not going to be caught by fifty thousand either.
Variables ¶
var ErrNotFound = errors.New("msgstore: message not found")
ErrNotFound means no such message.
Functions ¶
func DescribeHour ¶
DescribeHour renders an hour index as something a person reads.
func HourOfWeek ¶
HourOfWeek returns the rhythm index for a time. Monday 00:00 UTC is 0.
UTC deliberately, and it is a real limitation rather than an oversight: a clinic in a timezone with daylight saving has two hours a year that do not line up. Recorded here because the alternative - carrying a timezone per channel - is worth doing and is not done yet, and the wrong version of this silently shifts every clinic's evening by an hour twice a year.
func IdentityKinds ¶ added in v0.1.1
IdentityKinds lists the kinds of identifier that can be searched.
Sorted, because a Go map ranges randomly and a list of search options that reorders on every page load looks broken.
func NormaliseIdentifiers ¶ added in v0.1.18
NormaliseIdentifiers reduces identifiers (system|value, or a bare value) the way the identity index stores them.
Types ¶
type AttachmentInfo ¶
type AttachmentInfo struct {
Digest attach.Digest `json:"digest"`
Size int `json:"size"`
FirstSeen time.Time `json:"firstSeen"`
// Messages is how many stored messages refer to this payload. The number that makes deduplication visible: a
// document referenced forty times was stored once and would otherwise have been stored forty times.
Messages int `json:"messages"`
}
AttachmentInfo describes a stored payload without loading it.
type AttachmentStats ¶
type AttachmentStats struct {
Count int `json:"count"`
Bytes int64 `json:"bytes"`
// References is the total number of message-to-payload references. Reported alongside Count because the gap
// between them is the saving, and Count alone does not show it.
References int `json:"references"`
// Orphans are payloads no stored message refers to. Non-zero is not an error - a message can be pruned before
// the next sweep - but a number that keeps growing means the sweep is not running.
Orphans int `json:"orphans"`
}
AttachmentStats summarises the attachment store.
type Bucket ¶
type Bucket struct {
Start time.Time `json:"start"`
Total int64 `json:"total"`
Delivered int64 `json:"delivered"`
Filtered int64 `json:"filtered"`
Failed int64 `json:"failed"`
Unparseable int64 `json:"unparseable"`
}
Bucket is one point on a throughput chart.
type Delivery ¶
type Delivery struct {
Destination string `json:"destination"`
Status DeliveryStatus `json:"status"`
Attempts int `json:"attempts"`
DurationMS int64 `json:"durationMs"`
Error string `json:"error,omitempty"`
}
Delivery is the outcome at one destination.
type DeliveryStatus ¶
type DeliveryStatus string
DeliveryStatus is what happened at one destination.
const ( DeliveryDelivered DeliveryStatus = "delivered" DeliveryFailed DeliveryStatus = "failed" DeliveryFiltered DeliveryStatus = "filtered" )
Delivery statuses.
type DestinationStats ¶
type DestinationStats struct {
Destination string `json:"destination"`
Delivered int64 `json:"delivered"`
Failed int64 `json:"failed"`
Filtered int64 `json:"filtered"`
AvgAttempts float64 `json:"avgAttempts"`
AvgDuration float64 `json:"avgDurationMs"`
}
DestinationStats summarises per-destination delivery outcomes.
type ExpressionResult ¶
type ExpressionResult struct {
// Matches are the messages the expression accepted, newest first.
Matches []Message `json:"matches"`
// Examined is how many messages were parsed and tested.
Examined int `json:"examined"`
// Available is how many matched the narrowing criteria, so a reader can see what fraction was
// looked at.
Available int `json:"available"`
// Unreadable counts messages that could not be parsed and so could not be tested.
Unreadable int `json:"unreadable"`
// Truncated says the examine bound was reached, so there may be matches that were not looked
// at. Without this a partial answer reads as a complete one.
Truncated bool `json:"truncated"`
// Paths lists what the expression looked at, which is a useful check that it says what its
// author meant.
Paths []string `json:"paths,omitempty"`
}
ExpressionResult is what a content search found.
type ExpressionSearch ¶
type ExpressionSearch struct {
// TenantID is whose messages are searched. Like every other query here it is unconditional: empty means the default
// tenant, never every tenant.
TenantID string
// Where is a filter expression, in the same language channel filters use.
Where string
// Narrowing, applied in SQL before anything is parsed. Using these makes the difference
// between examining a thousand messages and examining a hundred thousand.
Channel string
MessageType string
Outcome Outcome
Since time.Time
Until time.Time
// Examine bounds how many messages are parsed. Zero uses a default.
Examine int
// Limit bounds how many matches are returned. Zero uses a default.
Limit int
}
ExpressionSearch describes a search by message content.
type Flow ¶
type Flow struct {
Since time.Time `json:"since"`
Bucket time.Duration `json:"bucketNanos"`
Channels []FlowChannel `json:"channels"`
}
Flow is the whole map over one window.
type FlowChannel ¶
type FlowChannel struct {
Channel string `json:"channel"`
// Received is the source edge - what arrived at this channel, before any destination.
Received []Bucket `json:"received"`
// Strands are the destination edges, sorted by name.
Strands []Strand `json:"strands"`
}
FlowChannel is one channel: what it received, and where it sent.
type HourStat ¶
type HourStat struct {
// Hour is the index, 0-167. Monday 00:00 is 0.
Hour int `json:"hour"`
// Median is the typical count for this hour.
Median float64 `json:"median"`
// Low and High are the smallest and largest counts seen.
//
// Kept because the spread is what tells somebody whether the median means anything. A feed that has carried
// between 2 and 900 messages in this hour has no useful normal, and the alert should say so rather than pick a
// number from the middle and imply confidence.
Low int64 `json:"low"`
High int64 `json:"high"`
// Floor is the level this hour normally stays above: the 20th percentile of what has been seen.
//
// This, not Low, is what decides whether an alert is safe. A shortfall alert fires when traffic is low, so the
// question is how low the feed goes on an ordinary week - if it routinely dips to a fifth of its median, an alert
// pitched anywhere useful will fire on ordinary weeks. The true minimum cannot answer that, because one bank
// holiday drags it to zero and would disqualify a feed that is otherwise perfectly steady.
Floor float64 `json:"floor"`
// Observations is how many weeks contributed a count for this hour.
Observations int `json:"observations"`
// Silent is how many of those observations were zero.
//
// The distinction that matters most. An hour that has been silent nine weeks out of ten is a quiet hour, and a
// tenth silent week is not news. An hour that has never been silent going quiet is the outage nobody would
// otherwise hear about, because silence raises no error.
Silent int `json:"silent"`
}
HourStat is what one hour of the week normally carries.
func (HourStat) Quiet ¶
Quiet reports whether this hour is normally silent.
The test is whether silence has ever been normal here, not whether it is common. An hour that is usually busy but was silent once - a bank holiday, a maintenance window - should not have that one week treated as licence to be silent again unnoticed. Being wrong in this direction produces an alert somebody dismisses; being wrong in the other produces a lab feed that stopped on Friday and was noticed on Monday.
func (HourStat) Reliable ¶
Reliable reports whether this hour's median is worth alerting on.
Fewer than two observations is not a pattern, and a median of zero cannot be fallen below. The third condition is the one that took two failing tests to get right: the feed must normally stay within a quarter of its median.
My first attempt compared the high against the median, and it was wrong in both directions at once. A feed that flushed a backlog once was disqualified for months afterwards, because a single 6,000-message hour against a normal 300 looks like chaos - it is not, it is one Tuesday. And a genuinely erratic feed carrying between 2 and 900 messages in the same hour every week passed, because its high was only three times its median. The high says nothing about whether an alert will cry wolf; a shortfall alert fires when traffic is low, so only the low end can answer that. Hence Floor, and hence the 20th percentile rather than the minimum.
type IdentityMatch ¶ added in v0.1.1
type IdentityMatch struct {
Message Message
// MatchedKind and MatchedValue are why this message is in the result.
//
// Returned because a result list of otherwise identical ADT messages gives no clue which
// field the search hit, and "it matched the account number, not the MRN" changes what the
// person does next.
MatchedKind identity.Kind
MatchedValue string
}
IdentityMatch is one message that matched, with the reason it matched.
type IdentityResult ¶ added in v0.1.1
type IdentityResult struct {
// Matches is initialised rather than nil so it marshals as an empty array. A client doing
// .length on null crashes on exactly the search that found nothing.
Matches []IdentityMatch
// Total is how many messages matched before the page was taken.
Total int
// Indexed reports whether identity indexing is switched on.
//
// Returned on every search, because the difference between "no message mentions that
// number" and "nothing has been indexed" is the difference between a clinical answer and
// a configuration mistake. Without this a site that never enabled indexing sees an empty
// result and concludes the patient's messages were never received.
Indexed bool
}
IdentityResult is what an identity search found.
type IdentitySearch ¶ added in v0.1.1
type IdentitySearch struct {
// Term is what the person typed: an MRN, a name, a date of birth, an accession.
Term string
// Kind narrows to one sort of identifier. Empty searches all of them, which is the
// default because the person often does not know which sort of number they are holding.
Kind identity.Kind
// TenantID scopes the search. Empty means the main tenant, matching Query.
TenantID string
// Channel, Since and Until narrow the same way the message list does, so a search can be
// confined to one feed or one afternoon.
Channel string
Since time.Time
Until time.Time
Limit int
Offset int
}
IdentitySearch describes a search for a message by an identifier.
type LastDelivery ¶ added in v0.1.6
type LastDelivery struct {
Channel string `json:"channel"`
Destination string `json:"destination"`
Delivered *time.Time `json:"lastDelivered,omitempty"`
Failed *time.Time `json:"lastFailed,omitempty"`
Error string `json:"lastError,omitempty"`
}
LastDelivery is the most recent outcome of a destination, for the connections dashboard: when it last delivered, when it last failed, and the error that failure gave.
type Message ¶
type Message struct {
ID int64 `json:"id"`
Channel string `json:"channel"`
// TenantID is which organisation this message belongs to.
//
// On the message rather than derived from the channel name, because channel
// names are not unique across tenants - two clinics both having an "adt-in" is
// the first thing that happens - so a name could not identify an owner.
TenantID string `json:"tenantId,omitempty"`
ReceivedAt time.Time `json:"receivedAt"`
ControlID string `json:"controlId,omitempty"`
MessageType string `json:"messageType,omitempty"`
TriggerEvent string `json:"triggerEvent,omitempty"`
Sender string `json:"sender,omitempty"`
Remote string `json:"remote,omitempty"`
Outcome Outcome `json:"outcome"`
AckCode string `json:"ackCode,omitempty"`
Size int `json:"size"`
DurationMS int64 `json:"durationMs"`
Error string `json:"error,omitempty"`
Segments int `json:"segments,omitempty"`
// Raw is the original bytes, loaded only when a single message is fetched.
// Listing thousands of messages must not drag their payloads along.
Raw []byte `json:"raw,omitempty"`
// Deliveries is the per-destination outcome, loaded with a single message.
Deliveries []Delivery `json:"deliveries,omitempty"`
// Attachments are the payloads this message carries, loaded with a single
// message. Described rather than included: the interface needs to say "a 4 MB
// PDF" without streaming four megabytes to do it.
Attachments []AttachmentInfo `json:"attachments,omitempty"`
// AttachmentError explains why Raw could not be fully reassembled, empty when
// it could. Reported rather than failing the fetch, because a message with an
// unresolvable payload still records that something arrived.
AttachmentError string `json:"attachmentError,omitempty"`
}
Message is one recorded message.
type MinuteCount ¶ added in v0.1.23
MinuteCount is how many messages one channel received in one minute.
type Outcome ¶
type Outcome string
Outcome is what happened to a message. These mirror the engine's outcomes, because collapsing them here would lose the distinction the engine went to trouble to preserve.
const ( // Delivered means every destination that wanted it accepted it. Delivered Outcome = "delivered" // Filtered means a filter rejected it. Nothing was sent and nothing failed. Filtered Outcome = "filtered" // Partial means some destinations have it and some do not. Partial Outcome = "partial" // Failed means nothing accepted it. Failed Outcome = "failed" // Unparseable means the bytes were not an HL7 message. Unparseable Outcome = "unparseable" // Pending means processing has not finished, which is what an on_receipt // acknowledgement leaves behind briefly. Pending Outcome = "pending" )
Message outcomes.
type Query ¶
type Query struct {
// TenantID scopes the query to one organisation. Required in multi-tenant operation.
//
// A field on the query rather than a separate argument, so it travels with everything else and a caller cannot
// build a query that forgets it. Empty means the main tenant, which keeps single-tenant callers unchanged - and
// the alternative, empty meaning "all tenants", is the sort of default that turns one missed line into a
// disclosure of patient data.
TenantID string
Channel string
Outcome Outcome
MessageType string
TriggerEvent string
ControlID string
Sender string
// Search matches against the stored payload. It is a substring match, which is
// slow on a large table and is exactly what somebody needs during an incident
// when all they have is an accession number.
Search string
Since time.Time
Until time.Time
Limit int
Offset int
// contains filtered or unexported fields
}
Query filters a message list.
type RecentFilter ¶ added in v0.1.7
type RecentFilter struct {
// Channels limits it to these channels; empty is every channel.
Channels []string
// Types are message_type prefixes (ORU, 999, 277, CT): a message matches when its type starts with one, or for X12
// contains it among its transaction sets. Empty is every type.
Types []string
// Outcomes limits it to these outcomes; empty is every outcome.
Outcomes []string
// WithRaw reads the payload too.
WithRaw bool
// Limit caps the rows; 0 is 5000.
Limit int
}
RecentFilter narrows Recent.
type RecentMessage ¶ added in v0.1.7
type RecentMessage struct {
ID int64
Channel string
Type string
Event string
Outcome Outcome
AckCode string
Sender string
DurationMS int64
ReceivedAt time.Time
// FailedDeliveries counts destinations that failed or are still queued.
FailedDeliveries int
Raw []byte
}
RecentMessage is one message, as much of it as a figure needs.
type Recorder ¶
Recorder adapts the store to the engine's Recorder interface.
The engine reports what happened and does not know about storage. This is the piece that decides to keep it, which is why a deployment that must not hold clinical content simply does not install one.
func NewRecorder ¶
NewRecorder builds a recorder.
func (*Recorder) RecordMessage ¶
func (r *Recorder) RecordMessage(ctx context.Context, record engine.MessageRecord)
RecordMessage stores one message record.
A storage failure is logged and swallowed on purpose. By the time this is called the message has already been delivered and acknowledged; failing here would either lose the acknowledgement or make the sender resend something that arrived. The log is the right place for it, and it is logged at error level so it is not invisible.
type Rhythm ¶
type Rhythm struct {
// Channel is the feed this describes.
Channel string `json:"channel"`
// Hours holds one entry per hour of the week, indexed 0-167, Monday 00:00 UTC at zero.
//
// Every hour is present even with no observations, so a caller cannot mistake a gap in the array for a quiet
// hour. Which of the two it is is the whole question, and Observations is what says.
Hours [HoursInWeek]HourStat `json:"hours"`
// Weeks is how many distinct calendar weeks contributed.
//
// Reported because it is the honest measure of how much this can be trusted. One week of history means every
// hour has exactly one observation and the median is that single number, which is not a rhythm - it is last
// Tuesday.
Weeks int `json:"weeks"`
// From and Until bound the history examined.
From time.Time `json:"from"`
Until time.Time `json:"until"`
}
Rhythm is what a channel's traffic normally looks like, hour by hour, across a week.
Why a week and not a day ¶
A fixed no-traffic threshold only works on a feed that never goes quiet. A hospital ADT feed is like that. Almost nothing in a clinic is: the lab feed stops at six, the referral feed is dead all weekend, and the immunisation registry submits once a night. Set a threshold those can pass at 3am on a Sunday and it will never catch a real outage on Tuesday afternoon. Set one that catches Tuesday and it pages every Saturday, so somebody turns it off, and then Tuesday is unwatched too.
So the shape being learned is the hour of the week - 168 of them - because that is the smallest unit that distinguishes Tuesday 2pm from Sunday 2am. A day-of-week-only model cannot tell 9am from midnight; an hour-only model cannot tell Wednesday from Sunday.
Why the median and not the mean ¶
One day where a backlog flushed 40,000 messages through a feed that normally carries 300 moves a mean far enough that the feed can then go completely silent without falling below it. The median ignores it. Feeds get backlogs, reprocessing runs and migration dumps often enough that this is not a hypothetical.
What this deliberately is not ¶
Not a forecast, and not anomaly detection in the machine-learning sense. It answers one question: for this hour of this weekday, what has this channel usually done? Everything clever beyond that would be harder to explain to somebody at 2am deciding whether to wake a colleague, and an alert nobody trusts is an alert nobody acts on.
func (*Rhythm) Describe ¶
Describe explains an hour in a sentence, for an alert somebody reads at 2am.
func (*Rhythm) Shortfall ¶
Shortfall reports how far below its normal an hour has fallen, as a fraction.
Zero means at or above normal. One means completely silent in an hour that is never silent. A caller alerting at 0.8 is asking to hear when a feed has lost four fifths of its traffic.
Returns false when nothing can honestly be said: an hour with no reliable history, or one where silence is normal. A rule that guessed here would fire on every quiet Sunday, get switched off, and take the useful alerts with it.
type Stats ¶
type Stats struct {
Total int64 `json:"total"`
ByOutcome map[string]int64 `json:"byOutcome"`
ByChannel map[string]int64 `json:"byChannel"`
ByType map[string]int64 `json:"byType"`
// FailedByChannel counts the failed and unparseable messages per channel, for the interface analyst's worst-first list.
FailedByChannel map[string]int64 `json:"failedByChannel"`
// RejectedBySender counts the messages this server answered with a negative acknowledgement (AE, AR, CE, CR), per
// sending application, so a partner sending bad messages shows as one.
RejectedBySender map[string]int64 `json:"rejectedBySender"`
OldestKept *time.Time `json:"oldestKept,omitempty"`
NewestKept *time.Time `json:"newestKept,omitempty"`
StoredBytes int64 `json:"storedBytes"`
}
Stats summarises activity for a dashboard.
type Store ¶
type Store struct {
// RetentionDays bounds how long messages are kept. Zero means forever, which
// is a choice an operator should have to make deliberately rather than get by
// default.
RetentionDays int
// StorePayloads controls whether the original bytes are kept. Turning it off
// keeps counts and outcomes while storing no clinical content, which is what
// a deployment that cannot hold PHI at rest needs.
StorePayloads bool
// IndexIdentity controls whether identifiers are extracted from payloads into a
// searchable index, so a message can be found by MRN or by name whatever format it
// arrived in.
//
// This is a privacy decision and not only a performance one. The identifiers are already
// in this database inside the payload column, so indexing them adds no data - but it does
// make them enumerable, and a database that could previously only be grepped for a
// number somebody already had can now be asked to list patients. That is a decision for
// the site.
//
// Independent of StorePayloads except in one direction, enforced in indexIdentity below:
// an installation that keeps no payloads is not given an identity index either, because
// there the extraction would be retaining clinical content that the site chose not to
// retain.
IndexIdentity bool
// PayloadDays is how long message bodies are kept, when set.
//
// Zero means half the retention window, which is what this did before the value
// was configurable - so an installation that says nothing keeps behaving as it did.
PayloadDays int
// RetentionDaysFn, StorePayloadsFn and PayloadDaysFn supersede the fields above when set.
//
// They exist so these can be changed while the server runs. Reading the fields
// directly while something else writes them is a data race, and the settings
// store already holds a lock around its values - so the safe arrangement is to
// ask it each time rather than copy the value in at startup.
//
// Assigned once before serving begins, like the fields they replace. It is the
// values behind them that change, not the functions.
RetentionDaysFn func() int
StorePayloadsFn func() bool
PayloadDaysFn func() int
IndexIdentityFn func() bool
// contains filtered or unexported fields
}
Store persists messages.
func NewStore ¶
NewStore prepares the schema on an existing database handle. Queries and writes share it.
func NewStoreWithReader ¶
NewStoreWithReader prepares the schema and uses a separate pool for queries.
Pass the read handle from internal/sqlitedb. Without it everything still works; reports just compete with delivery for the single write connection, which is measurably bad once the message table is large.
func (*Store) Attachment ¶
Attachment fetches a payload by digest.
func (*Store) AttachmentLookup ¶
AttachmentLookup returns a lookup bound to this store, for reassembly.
Errors are folded into "not found", because the caller's only correct response to either is to refuse the delivery, and attach.Reassemble already produces an error that says exactly that. A separate error path would mean two ways of describing the same outcome.
func (*Store) AttachmentStats ¶
func (s *Store) AttachmentStats(ctx context.Context) (AttachmentStats, error)
AttachmentStats reports what the attachment store holds.
func (*Store) Channels ¶
Channels lists the channels that have recorded messages, for filter dropdowns.
func (*Store) CountPayloads ¶
CountPayloads reports how many stored messages a channel has bodies for.
Shown next to a replay so somebody can see what fraction of their history was examined. A report over the last thousand of two hundred thousand messages is a different kind of evidence from one over the last thousand of eleven hundred.
func (*Store) CountRecords ¶
CountRecords counts messages recorded for a channel, with or without their bodies.
Paired with CountPayloads so a caller can tell "nothing has arrived" from "things arrived and their bodies were not kept". Those need different answers and look identical from a count of payloads alone: replay reported "no messages have been recorded for this channel yet" to somebody who had forty thousand of them and payload storage switched off.
That is a bad answer in a specific way. It sends them to look at the feed, which is working, instead of at a setting.
func (*Store) DestinationStats ¶
func (s *Store) DestinationStats(ctx context.Context, tenantID string, since time.Time) ([]DestinationStats, error)
DestinationStats returns per-destination counts since a time.
func (*Store) EraseMessages ¶ added in v0.1.18
EraseMessages deletes messages whole: the record, its body, its deliveries, its identifiers and attachments no other message uses. It answers how many were deleted.
func (*Store) Flow ¶
func (s *Store) Flow(ctx context.Context, tenantID string, since time.Time, bucket time.Duration) (*Flow, error)
Flow returns per-channel and per-destination series over a window.
One query for the destination edges rather than one per strand. A site with twenty channels and three destinations each would otherwise issue sixty queries to draw one screen, and the screen polls.
func (*Store) Get ¶
Get loads one message with its payload and deliveries. Get fetches one message, scoped to a tenant.
The tenant is a required argument rather than an optional filter, because identifiers are sequential and guessing one takes no skill at all. A signature that allowed it to be omitted would make the unsafe call the shorter one.
A message belonging to another tenant reports as not found rather than forbidden: telling a caller that a record exists but is not theirs is itself a disclosure.
func (*Store) IndexesIdentity ¶ added in v0.1.18
IndexesIdentity says whether messages can be found by identifier at all.
func (*Store) LastDeliveries ¶ added in v0.1.6
func (s *Store) LastDeliveries(ctx context.Context, tenantID string, since time.Time) ([]LastDelivery, error)
LastDeliveries reports, for every channel and destination with traffic since a time, its last delivery and last failure. It carries no message content: the error is the transport's, the times are the message's arrival.
func (*Store) LearnRhythm ¶
func (s *Store) LearnRhythm(ctx context.Context, tenantID string, weeks int) (map[string]*Rhythm, error)
LearnRhythm builds the weekly shape of every channel's traffic.
weeks bounds how far back to look. Four is a reasonable default: enough that one odd week cannot dominate the median, short enough that a feed whose volume genuinely changed - a new clinic site, a retired interface - is re-learned within a month rather than being compared against a shape that no longer exists.
func (*Store) LinkAttachments ¶
func (s *Store) LinkAttachments(ctx context.Context, messageID int64, digests []attach.Digest) error
LinkAttachments records which payloads a message refers to.
Recorded explicitly rather than derived by scanning stored messages for tokens. A message's raw bytes are dropped at half the retention window, so a scan would stop finding references while the message still exists - and the sweep would then delete a payload that a queued delivery still needs.
func (*Store) MessageAttachments ¶
MessageAttachments lists what one message refers to.
func (*Store) MinuteCounts ¶ added in v0.1.23
func (s *Store) MinuteCounts(ctx context.Context, tenantID string, since time.Time) ([]MinuteCount, error)
MinuteCounts returns each channel's messages per minute since a time, for learning what a channel normally carries.
Counts only, never the messages: the access monitor learns rates and hours, and has no use for content.
func (*Store) Prune ¶
Prune removes messages older than the retention window.
Content is removed before metadata, so a store under disk pressure loses the payloads first and keeps the record that a message existed. Losing the fact that something arrived is worse than losing its contents.
func (*Store) PutAttachment ¶
PutAttachment stores a payload, or does nothing if that exact payload is already held.
Deduplication is the point, so an insert that collides on the digest is success rather than an error. Two callers storing the same document concurrently is normal in a fan-out, and treating the loser as a failure would fail a delivery for having nothing to do.
func (*Store) Reader ¶ added in v0.1.18
Reader is the read pool, for a data protection search across stores.
func (*Store) Recent ¶ added in v0.1.7
func (s *Store) Recent(ctx context.Context, tenantID string, since time.Time, f RecentFilter) ([]RecentMessage, error)
Recent lists messages received since a time, newest first.
func (*Store) RecentPayloads ¶
RecentPayloads returns the stored bodies for a channel, newest first.
Only messages that were actually parsed and acted on are returned. A message the engine could not read tells you nothing about whether a transformation change is safe, and including them would put noise in a report whose value depends on being believed.
func (*Store) SearchByExpression ¶
func (s *Store) SearchByExpression(ctx context.Context, q ExpressionSearch) (*ExpressionResult, error)
SearchByExpression finds stored messages whose content matches an expression.
func (*Store) SearchByIdentity ¶ added in v0.1.1
func (s *Store) SearchByIdentity(ctx context.Context, q IdentitySearch) (*IdentityResult, error)
SearchByIdentity finds messages whose extracted identifiers match a term.
The term is normalised the same way the stored values were, so a name written SMITH^JOHN in the message is found by typing "Smith, John" and an MRN written MRN-001234 is found by typing mrn001234. Matching is exact on the normalised form rather than a prefix or a substring: a substring match on a normalised identifier makes every short number match thousands of messages, which is not a search result anybody can use.
func (*Store) SetHolds ¶ added in v0.1.18
SetHolds tells the pruner which identifiers are under a legal hold: messages carrying one are not deleted by the retention window while the hold lasts. The function is asked each time the pruner runs.
func (*Store) StartPruner ¶
func (s *Store) StartPruner(ctx context.Context, every time.Duration, onResult func(payloads, rows int64, err error)) func()
StartPruner prunes on a schedule and returns a stop function.
func (*Store) StoredBytes ¶ added in v0.1.11
StoredBytes is how much message content one tenant holds, for its storage limit.
func (*Store) SubjectMessages ¶ added in v0.1.18
func (s *Store) SubjectMessages(ctx context.Context, tenantID string, norms []string, limit int) ([]SubjectMessage, error)
SubjectMessages lists the messages whose extracted identifiers include any of norms (NormaliseIdentifiers), oldest first.
func (*Store) SweepAttachments ¶
SweepAttachments deletes payloads no stored message refers to.
A sweep rather than reference counting. A count is one number that has to be correct across crashes, concurrent fan-out and partial transactions, and when it drifts the failure is either a leak nobody notices or a payload deleted while something still needs it. A sweep recomputes the truth every time and cannot drift.
Called from Prune, so it runs on the same schedule as message retention.
func (*Store) Throughput ¶
func (s *Store) Throughput(ctx context.Context, tenantID string, since time.Time, bucket time.Duration, channel string) ([]Bucket, error)
Throughput returns message counts bucketed over time.
Aggregating in SQL rather than loading rows and counting in Go is what keeps a chart cheap enough to poll. A dashboard that costs a table scan is a dashboard somebody turns off.
type Strand ¶
type Strand struct {
Channel string `json:"channel"`
Destination string `json:"destination"`
// Buckets covers the full window, zero-filled.
Buckets []Bucket `json:"buckets"`
}
Strand is one channel-to-destination edge over time.
type SubjectMessage ¶ added in v0.1.18
type SubjectMessage struct {
ID int64 `json:"id"`
Channel string `json:"channel"`
ReceivedAt time.Time `json:"receivedAt"`
ControlID string `json:"controlId,omitempty"`
Type string `json:"type,omitempty"`
// Matched is the identifier it was found by, as the index holds it.
Matched string `json:"matched"`
}
SubjectMessage is one message about the subject.