mpi

package
v0.1.23 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package mpi is a master patient index: it takes patient records from every source system (each identified by its assigning authority and number), decides which belong to the same person, and gives each person an enterprise identifier (EUID). Certain matches link on their own; uncertain ones wait in a review queue for a person to decide; every link, unlink, merge and decision is recorded with who made it.

Matching is deterministic first (an identifier from a system the site trusts to be unique per person, such as a national identifier or an enterprise number) and probabilistic otherwise: Fellegi-Sunter weights over name, birth date, sex, postal code, phone and address, with Jaro-Winkler for names. It decides nothing clinical: it says which records are the same person, with its reasons, for a person to confirm when it is not sure.

Index

Constants

View Source
const DefaultTenant = "main"

DefaultTenant is the tenant of a single-tenant server, as the account store names it.

View Source
const MinEstimatePairs = 200

MinEstimatePairs is the fewest candidate pairs an estimate is made from. Fewer, and the fit says more about the handful of records than about the site.

Variables

View Source
var DefaultWeights = map[string]Weights{
	"family":     {M: 0.95, U: 0.01},
	"given":      {M: 0.90, U: 0.02},
	"birth_date": {M: 0.97, U: 0.0003},
	"sex":        {M: 0.98, U: 0.5},
	"postal":     {M: 0.85, U: 0.05},
	"phone":      {M: 0.80, U: 0.0001},
	"line":       {M: 0.80, U: 0.001},
}

DefaultWeights are reasonable starting values from the record-linkage literature for hospital registration data. A site with its own linked sample can measure better ones.

View Source
var ErrNotFound = errors.New("mpi: not found")

ErrNotFound is a record, person or review that is not there.

View Source
var ErrTooFewPairs = errors.New("mpi: too few candidate pairs to estimate weights from; register more records first")

ErrTooFewPairs means the index holds too little to estimate from yet.

Functions

func TenantOf

func TenantOf(id string) string

TenantOf returns id, or DefaultTenant when it is empty: a single-tenant channel has no tenant set, and its session's is the default.

func ToFHIR

func ToFHIR(p *Person, euidSystem string) map[string]any

ToFHIR writes a person as a FHIR Patient: the best-known demographics and every identifier, the EUID first.

Types

type Comparison

type Comparison struct {
	Score  float64       `json:"score"` // 0 to 1: the evidence found as a share of the evidence possible
	Fields []FieldResult `json:"fields"`
	// Safe is whether the pair may be linked without a person: family name and birth date both agree.
	Safe bool `json:"safe"`
}

Comparison is a scored pair of records.

func Compare

func Compare(a, b *Record, weights map[string]Weights) Comparison

Compare scores two records' demographics. Only fields present on both count, so a sparse record is judged on what it has.

type Counts

type Counts struct {
	Persons int `json:"persons"`
	Records int `json:"records"`
	Reviews int `json:"reviews"`
	// Linked is how many persons have more than one record: the duplicates the index has joined.
	Linked int `json:"linked"`
}

Counts are the index's size, for the console.

type Estimate

type Estimate struct {
	// Weights are the m and u probabilities per field, ready for Settings.Weights.
	Weights map[string]Weights `json:"weights"`
	// Compared is how many pairs had each field on both sides: a field compared on few pairs is estimated loosely.
	Compared map[string]int `json:"compared"`
	// Pairs is the number of candidate pairs the estimate was made from.
	Pairs int `json:"pairs"`
	// MatchShare is the estimated share of those pairs that are the same person.
	MatchShare float64 `json:"matchShare"`
	// Iterations and Converged describe the fit.
	Iterations int  `json:"iterations"`
	Converged  bool `json:"converged"`
}

Estimate is what EstimateWeights found in a site's own records.

func EstimateWeights

func EstimateWeights(pairs []Comparison) (Estimate, error)

EstimateWeights fits the Fellegi-Sunter m and u probabilities to unlabelled candidate pairs with the expectation-maximisation algorithm (Winkler 1988; Jaro 1989): each pair is a mixture of "same person" and "not", and the fit finds the probabilities under which the agreement seen is likeliest. No pair needs to have been reviewed.

Each field is agree (partial agreement counts as agreement) or disagree, and a field missing on either side is left out of that pair, as Compare leaves it out of the score. The usual conditional-independence assumption applies: names, date and address are treated as independent given the class, which they are not quite, so the m probabilities come out somewhat high. That is why an estimate is shown to an administrator and applied only when they choose.

func EstimateWeightsWithChance

func EstimateWeightsWithChance(pairs, random []Comparison) (Estimate, error)

EstimateWeightsWithChance is EstimateWeights with u fixed from pairs drawn at random, nearly all of them different people: how often each field agrees by chance in the whole index. Candidate pairs alone cannot tell it, because blocking chose them for agreeing on a name or a date, so their chance agreement on those fields is far higher than the index's. This is the usual remedy (as in Splink and fastLink): u from random pairs, m and the match share from EM.

type Event

type Event struct {
	At        time.Time `json:"at"`
	Actor     string    `json:"actor"`
	Action    string    `json:"action"`
	RecordID  int64     `json:"recordId,omitempty"`
	EUID      string    `json:"euid,omitempty"`
	OtherEUID string    `json:"otherEuid,omitempty"`
	Detail    string    `json:"detail,omitempty"`
}

Event is one entry in the index's history.

type Extracted

type Extracted struct {
	Record Record
	// MergeFrom is the prior identifier an A40 (or A34, A18) merges into the record, as system|value.
	MergeFrom string
}

Extracted is what a message says about a patient: the record, and a merge it asks for.

func FromFHIR

func FromFHIR(raw []byte) (*Extracted, error)

FromFHIR reads a FHIR Patient resource.

func FromHL7

func FromHL7(raw []byte, o FromOptions) (*Extracted, error)

FromHL7 reads the patient from an HL7 v2 message's PID (and MRG for a merge).

type FieldResult

type FieldResult struct {
	Field  string  `json:"field"`
	Result string  `json:"result"` // agree, partial, disagree, missing
	Weight float64 `json:"weight"`
}

FieldResult is how one field compared, for the explanation shown to a reviewer.

type FromOptions

type FromOptions struct {
	// Authorities maps an HL7 assigning authority's namespace (PID-3.4.1) to its system URI, for authorities that send no
	// universal identifier. Unmapped ones become urn:perfuse:aa:<namespace>.
	Authorities map[string]string
	// IDType is the identifier type code (PID-3.5) of the record's own number. Default MR; the first identifier when none has it.
	IDType string
	// SSNSystem names PID-19. Default http://hl7.org/fhir/sid/us-ssn.
	SSNSystem string
}

FromOptions says how to read a source system's identifiers.

type Match

type Match struct {
	Person *Person    `json:"person"`
	Score  float64    `json:"score"`
	Grade  string     `json:"grade"` // certain, probable, possible (FHIR's match-grade)
	Why    Comparison `json:"why"`
}

Match is one person found for a query, with how sure the index is.

type Outcome

type Outcome struct {
	Record *Record `json:"record"`
	// Action is linked (to an existing person, How saying why), new (a new person), updated (a known record's
	// demographics), or review (a new person, and a possible match queued for a person to decide).
	Action string  `json:"action"`
	How    string  `json:"how,omitempty"` // identifier, score
	Score  float64 `json:"score,omitempty"`
	// ReviewID is the queued decision, when there is one.
	ReviewID int64 `json:"reviewId,omitempty"`
	// Warnings are national numbers whose check digit failed (masked): kept on the record, never used to link it.
	Warnings []string `json:"warnings,omitempty"`
}

Outcome is what registering a record did.

type Person

type Person struct {
	EUID    string    `json:"euid"`
	Records []*Record `json:"records"`
	Created time.Time `json:"created"`
}

Person is one enterprise identity and every record linked to it.

func (*Person) Golden

func (p *Person) Golden() Record

Golden is the person's best-known demographics: the most recently updated value of each field.

func (*Person) Identifiers

func (p *Person) Identifiers(euidSystem string) []string

Identifiers lists every identifier a person has across its records, the EUID first.

type Record

type Record struct {
	ID int64 `json:"id"`
	// System is the assigning authority (a URI or OID) and Value the number it gave: what makes the record unique.
	System string `json:"system"`
	Value  string `json:"value"`

	Family    string `json:"family,omitempty"`
	Given     string `json:"given,omitempty"`
	BirthDate string `json:"birthDate,omitempty"` // YYYY-MM-DD, or digits
	Sex       string `json:"sex,omitempty"`       // M, F, O, U
	Phone     string `json:"phone,omitempty"`
	Line      string `json:"line,omitempty"`
	City      string `json:"city,omitempty"`
	Postal    string `json:"postal,omitempty"`
	// Identifiers are other identifiers the record carries (system|value): an enterprise number, a payer member number.
	Identifiers []string `json:"identifiers,omitempty"`
	// Source is where it came from, such as a channel name.
	Source string `json:"source,omitempty"`

	EUID      string    `json:"euid,omitempty"`
	UpdatedAt time.Time `json:"updatedAt"`
}

Record is one source system's record of a patient.

func (*Record) Key

func (r *Record) Key() string

Key is system|value.

type Review

type Review struct {
	ID        int64      `json:"id"`
	Record    *Record    `json:"record"`
	Candidate *Person    `json:"candidate"`
	Score     float64    `json:"score"`
	Why       Comparison `json:"why"`
	Status    string     `json:"status"` // open, same, different
	Created   time.Time  `json:"created"`
	DecidedBy string     `json:"decidedBy,omitempty"`
}

Review is a possible match waiting for a person.

type Settings

type Settings struct {
	// Automatic links a record to a person by score without asking anybody. Off by default: linking two people into one
	// (an overlay) puts one patient's results in another's chart, so until a site has watched the review queue and trusts
	// what it would have done, every scored match goes to a person. Trusted identifiers link either way.
	Automatic bool `json:"automatic"`
	// AutoLink is the score at or above which a pair links without review, when Automatic is on and name and birth date
	// also agree. Default 0.9. With Automatic off, matches above it are still queued, marked certain.
	AutoLink float64 `json:"autoLink"`
	// Review is the score at or above which a pair goes to the review queue. Default 0.6: low enough that twins and a sex
	// keyed wrongly reach a person, since only a person can tell those apart.
	Review float64 `json:"review"`
	// Trusted are identifier systems unique per person (a national identifier, an enterprise number): records sharing one
	// link at once. An identifier system shared by family members, such as a payer's subscriber number, must not be listed.
	Trusted []string `json:"trusted"`
	// EUIDSystem names the enterprise identifier in FHIR and PIX answers.
	EUIDSystem string `json:"euidSystem"`
	// Weights replaces DefaultWeights field by field, typically with values estimated from the site's own records
	// (EstimateWeights). Empty: the defaults.
	Weights map[string]Weights `json:"weights,omitempty"`
	// NationalSystems replaces a national identifier scheme's system URI (natid scheme ID to URI) where the country's guide
	// names it differently from Perfuse's default.
	NationalSystems map[string]string `json:"nationalSystems,omitempty"`
	// ShowNumbersTo is the lowest role that sees national numbers in full: viewer, operator, editor (the default) or admin.
	// Everybody below sees the last few characters only.
	ShowNumbersTo string `json:"showNumbersTo,omitempty"`
}

Settings are the matching thresholds and rules.

func (Settings) MaskedIdentifiers added in v0.1.18

func (s Settings) MaskedIdentifiers(list []string) []string

MaskedIdentifiers masks the national numbers in a list of system|value identifiers.

func (Settings) MaskedPerson added in v0.1.18

func (s Settings) MaskedPerson(p *Person) *Person

MaskedPerson is a copy of a person with every record's national numbers masked.

func (Settings) MaskedRecord added in v0.1.18

func (s Settings) MaskedRecord(r *Record) *Record

MaskedRecord is a copy of a record with its national numbers masked (all but the last few characters).

func (Settings) NumbersRole added in v0.1.18

func (s Settings) NumbersRole() string

NumbersRole is the lowest role that sees national numbers in full.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is the index, in a database of its own.

func Open

func Open(ctx context.Context, db *sql.DB) (*Store, error)

Open prepares the tables.

func (*Store) ByIdentifier

func (s *Store) ByIdentifier(ctx context.Context, tenant, ident string) (*Person, error)

ByIdentifier finds the person holding an identifier (system|value), on its record or as one of a record's identifiers.

func (*Store) CandidatePairs

func (s *Store) CandidatePairs(ctx context.Context, tenant string, max int) ([]Comparison, error)

CandidatePairs compares records the blocking would bring together (the same-sounding family name and birth year, the same birth date, the same phone), at most max pairs. These are the pairs the index actually decides on, linked or not, which is what the weights have to tell apart.

func (*Store) Count

func (s *Store) Count(ctx context.Context, tenant string) (Counts, error)

Count returns the index's size.

func (*Store) Decide

func (s *Store) Decide(ctx context.Context, tenant, actor string, reviewID int64, same bool) error

Decide closes a review: same links the record to the candidate person; different records that the two are not the same, so the pair is not offered again.

func (*Store) ErasePerson added in v0.1.18

func (s *Store) ErasePerson(ctx context.Context, tenant, actor, euid, why string) (records int, err error)

ErasePerson removes a person and every record linked to them: demographics, identifiers, reviews and the history of what happened to them. One event says an erasure happened, by whom, with no detail about the person.

func (*Store) EstimateFromRecords

func (s *Store) EstimateFromRecords(ctx context.Context, tenant string) (Estimate, error)

EstimateFromRecords estimates weights from the tenant's own records: u from random pairs, m from the candidate pairs.

func (*Store) Find

func (s *Store) Find(ctx context.Context, tenant string, q Record, set Settings, limit int) ([]Match, error)

Find scores the people who could be the person described, best first: what FHIR's Patient/$match and a demographic search answer. An identifier in the query that a person holds makes that person certain.

func (*Store) History

func (s *Store) History(ctx context.Context, tenant, euid string, limit int) ([]Event, error)

History lists what happened to a person, newest first: registrations, links, unlinks, merges and review decisions.

func (*Store) KnowsSystem added in v0.1.19

func (s *Store) KnowsSystem(ctx context.Context, tenant, system string) (bool, error)

KnowsSystem reports whether any record in the index has an identifier from the system: whether it is a patient identification domain the index manages, which an IHE query naming an unknown one must be told it is not.

func (s *Store) Link(ctx context.Context, tenant, actor string, recID int64, euid, why string) error

Link moves a record to a person by hand.

func (*Store) Merge

func (s *Store) Merge(ctx context.Context, tenant, actor, fromEUID, intoEUID, why string) error

Merge moves every record of one person into another, as an ADT merge (A40) or a reviewer would.

func (*Store) MergeIdentifiers

func (s *Store) MergeIdentifiers(ctx context.Context, tenant, actor, fromIdent, intoIdent, why string) (bool, error)

MergeIdentifiers merges the person holding one identifier into the person holding another, as an ADT A40 names them. An identifier the index does not hold is not an error: there is nothing to merge.

func (*Store) Person

func (s *Store) Person(ctx context.Context, tenant, euid string) (*Person, error)

Person reads one person by EUID.

func (*Store) RandomPairs

func (s *Store) RandomPairs(ctx context.Context, tenant string, n int) ([]Comparison, error)

RandomPairs compares n pairs of records drawn at random (deterministically, so an estimate can be repeated).

func (*Store) Reader added in v0.1.18

func (s *Store) Reader() *sql.DB

Reader is the index's database, for a data protection search across stores.

func (*Store) Recent

func (s *Store) Recent(ctx context.Context, tenant, q string, limit int) ([]*Person, error)

Recent lists the persons most recently changed, for the console's first view.

func (*Store) Record

func (s *Store) Record(ctx context.Context, tenant string, id int64) (*Record, error)

Record reads one record.

func (*Store) Register

func (s *Store) Register(ctx context.Context, tenant, actor string, in Record, set Settings) (*Outcome, error)

Register adds or updates a record and links it: to the person a trusted identifier names, to the best-scoring person when the match is certain, or to a new person, queueing a review when there is a possible match.

func (*Store) RegisterWithSettings

func (s *Store) RegisterWithSettings(ctx context.Context, tenant, actor string, r Record) (*Outcome, error)

RegisterWithSettings registers a record using the tenant's saved settings.

func (*Store) Reviews

func (s *Store) Reviews(ctx context.Context, tenant string, limit int) ([]Review, error)

Reviews lists the open reviews, highest score first.

func (*Store) SaveSettings

func (s *Store) SaveSettings(ctx context.Context, tenant, actor string, set Settings) error

SaveSettings stores a tenant's matching settings.

func (*Store) Settings

func (s *Store) Settings(ctx context.Context, tenant string) (Settings, error)

Settings reads the matching settings saved for a tenant, with defaults filled in.

func (*Store) SubjectPersons added in v0.1.18

func (s *Store) SubjectPersons(ctx context.Context, tenant string, idents []string, set Settings) ([]*Person, error)

SubjectPersons finds the people holding any of the identifiers (system|value), written as the index stores them, and the person named by an EUID when one of the identifiers is EUIDSystem|euid.

func (s *Store) Unlink(ctx context.Context, tenant, actor string, recID int64, why string) (string, error)

Unlink takes a record out of its person into a new one of its own, and remembers the two are not the same.

type Weights

type Weights struct {
	M float64 `json:"m"`
	U float64 `json:"u"`
}

Weights are the Fellegi-Sunter agreement probabilities for one field: M, that the field agrees when two records are the same person; U, that it agrees by chance when they are not. Agreement adds log2(M/U); disagreement adds log2((1-M)/(1-U)).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL