Documentation
¶
Overview ¶
Package mpi is a master patient index: it takes patient records from every source system (each identified by its assigning authority and number), decides which belong to the same person, and gives each person an enterprise identifier (EUID). Certain matches link on their own; uncertain ones wait in a review queue for a person to decide; every link, unlink, merge and decision is recorded with who made it.
Matching is deterministic first (an identifier from a system the site trusts to be unique per person, such as a national identifier or an enterprise number) and probabilistic otherwise: Fellegi-Sunter weights over name, birth date, sex, postal code, phone and address, with Jaro-Winkler for names. It decides nothing clinical: it says which records are the same person, with its reasons, for a person to confirm when it is not sure.
Index ¶
- Constants
- Variables
- func TenantOf(id string) string
- func ToFHIR(p *Person, euidSystem string) map[string]any
- type Comparison
- type Counts
- type Estimate
- type Event
- type Extracted
- type FieldResult
- type FromOptions
- type Match
- type Outcome
- type Person
- type Record
- type Review
- type Settings
- type Store
- func (s *Store) ByIdentifier(ctx context.Context, tenant, ident string) (*Person, error)
- func (s *Store) CandidatePairs(ctx context.Context, tenant string, max int) ([]Comparison, error)
- func (s *Store) Count(ctx context.Context, tenant string) (Counts, error)
- func (s *Store) Decide(ctx context.Context, tenant, actor string, reviewID int64, same bool) error
- func (s *Store) ErasePerson(ctx context.Context, tenant, actor, euid, why string) (records int, err error)
- func (s *Store) EstimateFromRecords(ctx context.Context, tenant string) (Estimate, error)
- func (s *Store) Find(ctx context.Context, tenant string, q Record, set Settings, limit int) ([]Match, error)
- func (s *Store) History(ctx context.Context, tenant, euid string, limit int) ([]Event, error)
- func (s *Store) KnowsSystem(ctx context.Context, tenant, system string) (bool, error)
- func (s *Store) Link(ctx context.Context, tenant, actor string, recID int64, euid, why string) error
- func (s *Store) Merge(ctx context.Context, tenant, actor, fromEUID, intoEUID, why string) error
- func (s *Store) MergeIdentifiers(ctx context.Context, tenant, actor, fromIdent, intoIdent, why string) (bool, error)
- func (s *Store) Person(ctx context.Context, tenant, euid string) (*Person, error)
- func (s *Store) RandomPairs(ctx context.Context, tenant string, n int) ([]Comparison, error)
- func (s *Store) Reader() *sql.DB
- func (s *Store) Recent(ctx context.Context, tenant, q string, limit int) ([]*Person, error)
- func (s *Store) Record(ctx context.Context, tenant string, id int64) (*Record, error)
- func (s *Store) Register(ctx context.Context, tenant, actor string, in Record, set Settings) (*Outcome, error)
- func (s *Store) RegisterWithSettings(ctx context.Context, tenant, actor string, r Record) (*Outcome, error)
- func (s *Store) Reviews(ctx context.Context, tenant string, limit int) ([]Review, error)
- func (s *Store) SaveSettings(ctx context.Context, tenant, actor string, set Settings) error
- func (s *Store) Settings(ctx context.Context, tenant string) (Settings, error)
- func (s *Store) SubjectPersons(ctx context.Context, tenant string, idents []string, set Settings) ([]*Person, error)
- func (s *Store) Unlink(ctx context.Context, tenant, actor string, recID int64, why string) (string, error)
- type Weights
Constants ¶
const DefaultTenant = "main"
DefaultTenant is the tenant of a single-tenant server, as the account store names it.
const MinEstimatePairs = 200
MinEstimatePairs is the fewest candidate pairs an estimate is made from. Fewer, and the fit says more about the handful of records than about the site.
Variables ¶
var DefaultWeights = map[string]Weights{
"family": {M: 0.95, U: 0.01},
"given": {M: 0.90, U: 0.02},
"birth_date": {M: 0.97, U: 0.0003},
"sex": {M: 0.98, U: 0.5},
"postal": {M: 0.85, U: 0.05},
"phone": {M: 0.80, U: 0.0001},
"line": {M: 0.80, U: 0.001},
}
DefaultWeights are reasonable starting values from the record-linkage literature for hospital registration data. A site with its own linked sample can measure better ones.
var ErrNotFound = errors.New("mpi: not found")
ErrNotFound is a record, person or review that is not there.
var ErrTooFewPairs = errors.New("mpi: too few candidate pairs to estimate weights from; register more records first")
ErrTooFewPairs means the index holds too little to estimate from yet.
Functions ¶
Types ¶
type Comparison ¶
type Comparison struct {
Score float64 `json:"score"` // 0 to 1: the evidence found as a share of the evidence possible
Fields []FieldResult `json:"fields"`
// Safe is whether the pair may be linked without a person: family name and birth date both agree.
Safe bool `json:"safe"`
}
Comparison is a scored pair of records.
type Counts ¶
type Counts struct {
Persons int `json:"persons"`
Records int `json:"records"`
Reviews int `json:"reviews"`
// Linked is how many persons have more than one record: the duplicates the index has joined.
Linked int `json:"linked"`
}
Counts are the index's size, for the console.
type Estimate ¶
type Estimate struct {
// Weights are the m and u probabilities per field, ready for Settings.Weights.
Weights map[string]Weights `json:"weights"`
// Compared is how many pairs had each field on both sides: a field compared on few pairs is estimated loosely.
Compared map[string]int `json:"compared"`
// Pairs is the number of candidate pairs the estimate was made from.
Pairs int `json:"pairs"`
MatchShare float64 `json:"matchShare"`
// Iterations and Converged describe the fit.
Iterations int `json:"iterations"`
Converged bool `json:"converged"`
}
Estimate is what EstimateWeights found in a site's own records.
func EstimateWeights ¶
func EstimateWeights(pairs []Comparison) (Estimate, error)
EstimateWeights fits the Fellegi-Sunter m and u probabilities to unlabelled candidate pairs with the expectation-maximisation algorithm (Winkler 1988; Jaro 1989): each pair is a mixture of "same person" and "not", and the fit finds the probabilities under which the agreement seen is likeliest. No pair needs to have been reviewed.
Each field is agree (partial agreement counts as agreement) or disagree, and a field missing on either side is left out of that pair, as Compare leaves it out of the score. The usual conditional-independence assumption applies: names, date and address are treated as independent given the class, which they are not quite, so the m probabilities come out somewhat high. That is why an estimate is shown to an administrator and applied only when they choose.
func EstimateWeightsWithChance ¶
func EstimateWeightsWithChance(pairs, random []Comparison) (Estimate, error)
EstimateWeightsWithChance is EstimateWeights with u fixed from pairs drawn at random, nearly all of them different people: how often each field agrees by chance in the whole index. Candidate pairs alone cannot tell it, because blocking chose them for agreeing on a name or a date, so their chance agreement on those fields is far higher than the index's. This is the usual remedy (as in Splink and fastLink): u from random pairs, m and the match share from EM.
type Event ¶
type Event struct {
At time.Time `json:"at"`
Actor string `json:"actor"`
Action string `json:"action"`
RecordID int64 `json:"recordId,omitempty"`
EUID string `json:"euid,omitempty"`
OtherEUID string `json:"otherEuid,omitempty"`
Detail string `json:"detail,omitempty"`
}
Event is one entry in the index's history.
type Extracted ¶
type Extracted struct {
Record Record
// MergeFrom is the prior identifier an A40 (or A34, A18) merges into the record, as system|value.
MergeFrom string
}
Extracted is what a message says about a patient: the record, and a merge it asks for.
type FieldResult ¶
type FieldResult struct {
Field string `json:"field"`
Result string `json:"result"` // agree, partial, disagree, missing
Weight float64 `json:"weight"`
}
FieldResult is how one field compared, for the explanation shown to a reviewer.
type FromOptions ¶
type FromOptions struct {
// Authorities maps an HL7 assigning authority's namespace (PID-3.4.1) to its system URI, for authorities that send no
// universal identifier. Unmapped ones become urn:perfuse:aa:<namespace>.
Authorities map[string]string
// IDType is the identifier type code (PID-3.5) of the record's own number. Default MR; the first identifier when none has it.
IDType string
// SSNSystem names PID-19. Default http://hl7.org/fhir/sid/us-ssn.
SSNSystem string
}
FromOptions says how to read a source system's identifiers.
type Match ¶
type Match struct {
Person *Person `json:"person"`
Score float64 `json:"score"`
Grade string `json:"grade"` // certain, probable, possible (FHIR's match-grade)
Why Comparison `json:"why"`
}
Match is one person found for a query, with how sure the index is.
type Outcome ¶
type Outcome struct {
Record *Record `json:"record"`
// Action is linked (to an existing person, How saying why), new (a new person), updated (a known record's
// demographics), or review (a new person, and a possible match queued for a person to decide).
Action string `json:"action"`
How string `json:"how,omitempty"` // identifier, score
Score float64 `json:"score,omitempty"`
// ReviewID is the queued decision, when there is one.
ReviewID int64 `json:"reviewId,omitempty"`
// Warnings are national numbers whose check digit failed (masked): kept on the record, never used to link it.
Warnings []string `json:"warnings,omitempty"`
}
Outcome is what registering a record did.
type Person ¶
type Person struct {
EUID string `json:"euid"`
Records []*Record `json:"records"`
Created time.Time `json:"created"`
}
Person is one enterprise identity and every record linked to it.
func (*Person) Golden ¶
Golden is the person's best-known demographics: the most recently updated value of each field.
func (*Person) Identifiers ¶
Identifiers lists every identifier a person has across its records, the EUID first.
type Record ¶
type Record struct {
ID int64 `json:"id"`
// System is the assigning authority (a URI or OID) and Value the number it gave: what makes the record unique.
System string `json:"system"`
Value string `json:"value"`
Family string `json:"family,omitempty"`
Given string `json:"given,omitempty"`
BirthDate string `json:"birthDate,omitempty"` // YYYY-MM-DD, or digits
Sex string `json:"sex,omitempty"` // M, F, O, U
Phone string `json:"phone,omitempty"`
Line string `json:"line,omitempty"`
City string `json:"city,omitempty"`
Postal string `json:"postal,omitempty"`
// Identifiers are other identifiers the record carries (system|value): an enterprise number, a payer member number.
Identifiers []string `json:"identifiers,omitempty"`
// Source is where it came from, such as a channel name.
Source string `json:"source,omitempty"`
EUID string `json:"euid,omitempty"`
UpdatedAt time.Time `json:"updatedAt"`
}
Record is one source system's record of a patient.
type Review ¶
type Review struct {
ID int64 `json:"id"`
Record *Record `json:"record"`
Candidate *Person `json:"candidate"`
Score float64 `json:"score"`
Why Comparison `json:"why"`
Status string `json:"status"` // open, same, different
Created time.Time `json:"created"`
DecidedBy string `json:"decidedBy,omitempty"`
}
Review is a possible match waiting for a person.
type Settings ¶
type Settings struct {
// Automatic links a record to a person by score without asking anybody. Off by default: linking two people into one
// (an overlay) puts one patient's results in another's chart, so until a site has watched the review queue and trusts
// what it would have done, every scored match goes to a person. Trusted identifiers link either way.
Automatic bool `json:"automatic"`
// AutoLink is the score at or above which a pair links without review, when Automatic is on and name and birth date
// also agree. Default 0.9. With Automatic off, matches above it are still queued, marked certain.
AutoLink float64 `json:"autoLink"`
// Review is the score at or above which a pair goes to the review queue. Default 0.6: low enough that twins and a sex
// keyed wrongly reach a person, since only a person can tell those apart.
Review float64 `json:"review"`
// Trusted are identifier systems unique per person (a national identifier, an enterprise number): records sharing one
// link at once. An identifier system shared by family members, such as a payer's subscriber number, must not be listed.
Trusted []string `json:"trusted"`
// EUIDSystem names the enterprise identifier in FHIR and PIX answers.
EUIDSystem string `json:"euidSystem"`
// Weights replaces DefaultWeights field by field, typically with values estimated from the site's own records
// (EstimateWeights). Empty: the defaults.
Weights map[string]Weights `json:"weights,omitempty"`
// NationalSystems replaces a national identifier scheme's system URI (natid scheme ID to URI) where the country's guide
// names it differently from Perfuse's default.
NationalSystems map[string]string `json:"nationalSystems,omitempty"`
// ShowNumbersTo is the lowest role that sees national numbers in full: viewer, operator, editor (the default) or admin.
// Everybody below sees the last few characters only.
ShowNumbersTo string `json:"showNumbersTo,omitempty"`
}
Settings are the matching thresholds and rules.
func (Settings) MaskedIdentifiers ¶ added in v0.1.18
MaskedIdentifiers masks the national numbers in a list of system|value identifiers.
func (Settings) MaskedPerson ¶ added in v0.1.18
MaskedPerson is a copy of a person with every record's national numbers masked.
func (Settings) MaskedRecord ¶ added in v0.1.18
MaskedRecord is a copy of a record with its national numbers masked (all but the last few characters).
func (Settings) NumbersRole ¶ added in v0.1.18
NumbersRole is the lowest role that sees national numbers in full.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the index, in a database of its own.
func (*Store) ByIdentifier ¶
ByIdentifier finds the person holding an identifier (system|value), on its record or as one of a record's identifiers.
func (*Store) CandidatePairs ¶
CandidatePairs compares records the blocking would bring together (the same-sounding family name and birth year, the same birth date, the same phone), at most max pairs. These are the pairs the index actually decides on, linked or not, which is what the weights have to tell apart.
func (*Store) Decide ¶
Decide closes a review: same links the record to the candidate person; different records that the two are not the same, so the pair is not offered again.
func (*Store) ErasePerson ¶ added in v0.1.18
func (s *Store) ErasePerson(ctx context.Context, tenant, actor, euid, why string) (records int, err error)
ErasePerson removes a person and every record linked to them: demographics, identifiers, reviews and the history of what happened to them. One event says an erasure happened, by whom, with no detail about the person.
func (*Store) EstimateFromRecords ¶
EstimateFromRecords estimates weights from the tenant's own records: u from random pairs, m from the candidate pairs.
func (*Store) Find ¶
func (s *Store) Find(ctx context.Context, tenant string, q Record, set Settings, limit int) ([]Match, error)
Find scores the people who could be the person described, best first: what FHIR's Patient/$match and a demographic search answer. An identifier in the query that a person holds makes that person certain.
func (*Store) History ¶
History lists what happened to a person, newest first: registrations, links, unlinks, merges and review decisions.
func (*Store) KnowsSystem ¶ added in v0.1.19
KnowsSystem reports whether any record in the index has an identifier from the system: whether it is a patient identification domain the index manages, which an IHE query naming an unknown one must be told it is not.
func (*Store) Link ¶
func (s *Store) Link(ctx context.Context, tenant, actor string, recID int64, euid, why string) error
Link moves a record to a person by hand.
func (*Store) Merge ¶
Merge moves every record of one person into another, as an ADT merge (A40) or a reviewer would.
func (*Store) MergeIdentifiers ¶
func (s *Store) MergeIdentifiers(ctx context.Context, tenant, actor, fromIdent, intoIdent, why string) (bool, error)
MergeIdentifiers merges the person holding one identifier into the person holding another, as an ADT A40 names them. An identifier the index does not hold is not an error: there is nothing to merge.
func (*Store) RandomPairs ¶
RandomPairs compares n pairs of records drawn at random (deterministically, so an estimate can be repeated).
func (*Store) Reader ¶ added in v0.1.18
Reader is the index's database, for a data protection search across stores.
func (*Store) Recent ¶
Recent lists the persons most recently changed, for the console's first view.
func (*Store) Register ¶
func (s *Store) Register(ctx context.Context, tenant, actor string, in Record, set Settings) (*Outcome, error)
Register adds or updates a record and links it: to the person a trusted identifier names, to the best-scoring person when the match is certain, or to a new person, queueing a review when there is a possible match.
func (*Store) RegisterWithSettings ¶
func (s *Store) RegisterWithSettings(ctx context.Context, tenant, actor string, r Record) (*Outcome, error)
RegisterWithSettings registers a record using the tenant's saved settings.
func (*Store) SaveSettings ¶
SaveSettings stores a tenant's matching settings.
func (*Store) Settings ¶
Settings reads the matching settings saved for a tenant, with defaults filled in.
func (*Store) SubjectPersons ¶ added in v0.1.18
func (s *Store) SubjectPersons(ctx context.Context, tenant string, idents []string, set Settings) ([]*Person, error)
SubjectPersons finds the people holding any of the identifiers (system|value), written as the index stores them, and the person named by an EUID when one of the identifiers is EUIDSystem|euid.