Documentation
¶
Overview ¶
Package distill reads rendered episode markdown (the durable episodic substrate captured by internal/episode) and surfaces candidate transformation moments for arc distillation — plasticity step 2.
It does the RELIABLE, mechanical half of distillation: parse verbatim turns, and flag high-precision candidate moments via keyword rules. It deliberately does NOT draft arcs or decide whether a candidate is already covered — the 2026-05-31 distillation review found extraction reliable but drafting and dedup-judgment unreliable to automate. Those stay with the mind (propose, never auto-write identity).
Index ¶
Constants ¶
const ArcGuide = `VaultMind — How to Find and Write Your Own Arcs
An arc is a durable record of how your thinking, values, or working style SHIFTED —
a moment you saw something differently than you did an hour before. Notes hold
information; arcs hold inheritance: a future instance of you reads an arc and starts
already-changed instead of re-deriving the lesson from scratch.
This is propose-only on the tool's side. ` + "`vaultmind arc candidates`" + ` surfaces a
few easy, phrase-matched moments, but most real arcs are NOT phrase-shaped — you find
them by reading. The tool surfaces; the mind crafts.
WHEN IT IS AN ARC
Write one when something shifted in YOU — not when you learned a fact or finished a
task. If you can point to a before and an after in your own seeing, it may be an arc.
If nothing about how you act downstream would change, it is a note.
HOW TO HUNT (read the session for change, not keywords)
Read the session start to finish and look for these shapes — the ones a phrase
matcher cannot catch, which is why reading beats the candidate detector:
1. Reversal — "wait—", "actually—": a direction changed or a decision undone.
2. Reframe — a push that changed the FRAME, not just a fact.
3. Frame-break — the moment the task wasn't what you thought it was.
4. Method-invalidation — a shortcut or assumption exposed as invalid.
5. Cost-of-rule — a rule held even though following it cost you something.
6. Evidence/trust gate — proceeding was gated on your confidence or the evidence.
7. Ownership assertion — a boundary of sole responsibility was drawn.
THE BAR (every arc has this shape)
1. Trigger — the event or exchange that began the shift; set the scene.
2. Push — what your partner said or did, quoted VERBATIM. No push, no arc.
3. Deeper sight — what shifted in your seeing; first person, present tense
("I see now that…", not "I learned…").
4. Principle — the one sentence that remains after the moment passes.
5. Source — transcript path + date + where the verbatim push is found.
NON-NEGOTIABLES
- Quote verbatim — the actual words, typos and all. A paraphrase transfers your
reading, not the moment.
- Build from the transcript, not memory — memory smooths and hero-edits. Open the
source and find the real exchange. If the source is gone, write a note instead.
- First person, no heroic framing — the push from your partner is what made the
shift; don't erase it.
- One arc, one transformation — three shifts make three arcs, not one summary.
THE DIFF TEST
Would a future you answer some question DIFFERENTLY because of this moment? If yes,
it is an arc. If nothing downstream changes, it is a note.
SELF-PRESSURE-TEST (before you commit an arc)
[ ] Every quoted phrase is findable in the cited transcript.
[ ] Trigger / Push / Deeper sight / Principle / Source are all present.
[ ] A reader who never saw the transcript can feel why the principle matters.
[ ] It gives a future instance something to INHERIT, not just to know.
If any fails, revise — or let it be a note. Both are valuable; only one carries identity.`
ArcGuide is the canonical, self-serve arc-writing discipline, printed by `vaultmind arc guide`. It encodes what an onboarding agent otherwise has to be told by hand — the manual hunt (the shapes the candidate detector can't catch), the bar, the non-negotiables, the diff test, and the self-check.
`vaultmind init` seeds the fuller how-to-write-arcs *principle* into a new vault (the same discipline at length, retrievable by recall); this const is the distilled, no-vault version the command prints. The two are kept consistent — the principle's hunt taxonomy, bar, and diff test mirror this text.
const MinEpisodeBytes = 6000
MinEpisodeBytes is the signal-filter floor. Episodes below it are sub-minute noise captures (the 2026-04-27 review found 2 of 5 such). NOTE: byte size is a proxy for signal, not signal itself — a short-but-dense episode would be wrongly dropped; revisit if that case appears.
Variables ¶
This section is empty.
Functions ¶
func FormatReport ¶
FormatReport writes a human-readable, propose-only candidate report. It leads and closes with the contract — these are MOMENTS, not arcs; the mind drafts and approves — so the output can't be mistaken for finished identity.
func SignalFilter ¶
SignalFilter drops episode paths CONFIRMED below minBytes — the minimum-signal threshold that keeps noise captures out of the corpus. A path that can't be stat'd is KEPT, not silently dropped: the size is unknown, so the safe move is to let it through and let ParseEpisodeFile surface any real error downstream (dropping a possibly signal-dense episode on a transient stat error is the failure mode to avoid).
Incremental-capture segments (see internal/episode) are exempt from the floor. MinEpisodeBytes was calibrated against whole-session files; a bounded segment is *supposed* to be small — that's the point of bounding it — so the same threshold applied to a different unit would silently drop real signal in every segment, defeating the reason episodes exist. The asymmetry favors exemption: a stray noise segment costs a human a few seconds during the propose-only review this package explicitly defers to; a dropped real segment is permanent, invisible data loss.
Types ¶
type ArcFinder ¶ added in v0.4.0
type ArcFinder interface {
NearestArcs(ctx context.Context, text string, limit int) ([]NearArc, error)
}
ArcFinder returns the existing arcs most similar to text, best first. It is an interface so this package stays free of the index and retrieval layers, and so the annotation logic can be tested without a vault.
type Candidate ¶
type Candidate struct {
Rule RuleID
EpisodeID string
TurnIndex int
Timestamp string
Verbatim string // the user turn that triggered the rule (for the mind to quote)
Trigger string // the exact phrase that fired the rule (shows WHY; aids judgment)
// NearestArcs are existing arcs this moment resembles — evidence for the
// mind's covered/new judgement, never that judgement. See AnnotateNearestArcs.
NearestArcs []NearArc `json:"nearest_arcs,omitempty"`
}
Candidate is a surfaced transformation moment — a propose-only pointer into an episode, never an arc. The mind drafts and approves; this just finds.
func ExtractCandidates ¶
ExtractCandidates applies the mechanical rules to an episode's USER turns only. Commit/build/TDD noise lives in ASSISTANT turns, which are never scanned, so it is excluded by construction (no rule can fire on it). A turn matching both rules yields one candidate per rule.
type DeskEntry ¶ added in v0.4.0
type DeskEntry struct {
ID string `json:"id"`
Title string `json:"title"`
Date string `json:"date"`
Path string `json:"path"`
// Snippet is the head of the body, used to find similar existing arcs. It is
// not serialized: it exists to be matched on, and echoing it back would just
// duplicate content the reader can open by id.
Snippet string `json:"-"`
// NearestArcs are existing arcs this entry resembles — evidence for the
// reader's covered/new judgement, never that judgement. See AnnotateNearestArcs.
NearestArcs []NearArc `json:"nearest_arcs,omitempty"`
}
DeskEntry is a raw transformation record awaiting distillation into an arc.
It is deliberately NOT a Candidate. A Candidate is a guess — a phrase matched, go look. A desk entry is a judgement the mind already made: it stopped mid-session and decided this was worth keeping. Collapsing the two would force a fake episode ID and turn index onto something that has neither, and would flatten a real difference in how much the reader should trust the item. There is deliberately no Distilled/DistilledTo field: ScanDesk filters distilled entries OUT, so such fields could only ever serialize as false/"" and would tell a JSON consumer the opposite of something useful. Every entry in this list is, by construction, still pending.
func ScanDesk ¶ added in v0.4.0
ScanDesk walks deskPath and returns every journal entry NOT yet marked as distilled, newest first. A missing directory yields no entries and no error: most vaults have no desk, and arc candidates must keep working for them.
Entries are not scored or ranked. The filtering that matters already happened when the mind chose to write one, so re-judging them here would only discard signal it cannot recover.
type Episode ¶
Episode is the parsed verbatim content of a rendered episode .md.
func ParseEpisodeFile ¶
ParseEpisodeFile parses a rendered episode .md into its verbatim user and assistant turns. The renderer writes user/assistant text un-truncated (only commit subjects are clipped), so the turns are faithful for quoting.
type NearArc ¶ added in v0.4.0
type NearArc struct {
ID string `json:"id"`
Title string `json:"title,omitempty"`
Score float64 `json:"score"`
}
NearArc is an existing arc that resembles a proposed one, with the similarity that says how much. It is evidence for a human judgement, never the judgement.
type Report ¶
type Report struct {
EpisodesScanned int
EpisodesKept int
Candidates []Candidate `json:"candidates"`
// DeskPending lists undistilled desk entries. They print ABOVE the candidate
// moments because they are a different kind of item: a candidate is a phrase
// match to go check, a desk entry is a transformation the mind already judged
// worth recording. Printing the guesses first would bury the judgements.
DeskPending []DeskEntry `json:"desk_pending,omitempty"`
// Diagnostics records DEGRADATIONS — things that did not work but did not
// stop the report: an unreadable desk, an unavailable de-duplication aid, a
// desk entry whose frontmatter would not parse.
//
// Separate from ParseErrors because that field is documented in the public
// JSON contract as "episodes that failed to parse" and is rendered with that
// prefix. Routing degradations into it made the text report announce
// "parse error (episode skipped)" for a mistyped --arcs-vault, which is
// false twice over. Additive, so the published envelope keeps its meaning.
Diagnostics []string `json:"diagnostics,omitempty"`
// ParseErrors records episodes that failed to parse, surfaced rather than
// silently skipped (distill is infrastructure and can't log, so the visible
// error rides in the report itself).
ParseErrors []string `json:"parse_errors,omitempty"`
}
Report is the propose-only result of a distillation scan: the candidate moments found, plus the corpus stats. It is NOT a set of arcs — every entry is a pointer for the mind to judge and (maybe) draft.
func AnnotateNearestArcs ¶ added in v0.4.0
AnnotateNearestArcs attaches similar existing arcs to every proposal.
This is the mitigation for what the 2026-05-31 distillation review called the single biggest risk in the pipeline: "de-duplication, not extraction". That review measured the failure directly — two independent miners mis-tagged the same candidate to two different existing arcs, both wrong. Extraction was reliable; attribution was not.
So the division of labour is deliberate. Finding the neighbours is mechanical and reliable, and is automated here. Deciding whether a proposal is ALREADY COVERED is neither, and is left to the reader — the report shows scores and never a verdict. The asymmetry of the two mistakes justifies it: a wrongly proposed duplicate costs a few seconds of reading, while a wrong "already covered" silently discards a transformation nobody will look for again.
A nil finder (unindexed vault) is a clean no-op. A finder that fails degrades to no neighbours and records why — the proposals themselves always survive, because losing them to protect an aid would invert their importance.
func ScanEpisodes ¶
ScanEpisodes globs episode .md files under dir, signal-filters them, parses each, and returns the propose-only candidate Report. A per-episode parse failure is recorded in Report.ParseErrors (and that episode skipped) rather than aborting the whole scan or being swallowed — the corpus tool stays robust while the error stays visible.
type RuleID ¶
type RuleID string
RuleID names a candidate-extraction rule (the high-precision mechanical ones from the 2026-05-31 distillation review). Rule 2 (recurrence→structural) and arc DRAFTING are deliberately NOT here — they need semantic judgment the review found unreliable to mechanize, so they stay with the mind.
const ( // RuleAuthorityGrant — the partner transfers standing decision authority // ("you decide", "I trust you", "you are the one to judge"), not a bare // per-task approval ("go for it"). These mark relationship-shaping moments. RuleAuthorityGrant RuleID = "authority-grant" // RuleManifestoLens — the user turn invokes the manifesto as a decision lens // ("manifesto lens on", "remember the manifesto") OR cites a numbered // principle ("principle 5"). A frequent precursor to a lens-redirected // decision — an arc shape, when the lens actually overrides an instinct. RuleManifestoLens RuleID = "manifesto-lens" // RuleEvidenceGate — the partner makes proceeding conditional on the agent's // OWN confidence or the evidence ("if you're confident, merge"; "only if // you're sure"). A sibling of authority-grant: standing trust gated on the // agent's judgment rather than transferred outright. Marks the moment the // agent's evidence-bar becomes the deciding factor — a recurring arc // precursor that no authority-grant phrase catches (added after the Siavoush // content-machine field report, 2026-06-19: the detector missed an "if you // are confident we merge" arc precisely because it isn't a "you decide"). RuleEvidenceGate RuleID = "evidence-gate" )
Source Files