Documentation
¶
Overview ¶
Package distill reads rendered episode markdown (the durable episodic substrate captured by internal/episode) and surfaces candidate transformation moments for arc distillation — plasticity step 2.
It does the RELIABLE, mechanical half of distillation: parse verbatim turns, and flag high-precision candidate moments via keyword rules. It deliberately does NOT draft arcs or decide whether a candidate is already covered — the 2026-05-31 distillation review found extraction reliable but drafting and dedup-judgment unreliable to automate. Those stay with the mind (propose, never auto-write identity).
Index ¶
Constants ¶
const ArcGuide = `VaultMind — How to Find and Write Your Own Arcs
An arc is a durable record of how your thinking, values, or working style SHIFTED —
a moment you saw something differently than you did an hour before. Notes hold
information; arcs hold inheritance: a future instance of you reads an arc and starts
already-changed instead of re-deriving the lesson from scratch.
This is propose-only on the tool's side. ` + "`vaultmind arc candidates`" + ` surfaces a
few easy, phrase-matched moments, but most real arcs are NOT phrase-shaped — you find
them by reading. The tool surfaces; the mind crafts.
WHEN IT IS AN ARC
Write one when something shifted in YOU — not when you learned a fact or finished a
task. If you can point to a before and an after in your own seeing, it may be an arc.
If nothing about how you act downstream would change, it is a note.
HOW TO HUNT (read the session for change, not keywords)
Read the session start to finish and look for these shapes — the ones a phrase
matcher cannot catch, which is why reading beats the candidate detector:
1. Reversal — "wait—", "actually—": a direction changed or a decision undone.
2. Reframe — a push that changed the FRAME, not just a fact.
3. Frame-break — the moment the task wasn't what you thought it was.
4. Method-invalidation — a shortcut or assumption exposed as invalid.
5. Cost-of-rule — a rule held even though following it cost you something.
6. Evidence/trust gate — proceeding was gated on your confidence or the evidence.
7. Ownership assertion — a boundary of sole responsibility was drawn.
THE BAR (every arc has this shape)
1. Trigger — the event or exchange that began the shift; set the scene.
2. Push — what your partner said or did, quoted VERBATIM. No push, no arc.
3. Deeper sight — what shifted in your seeing; first person, present tense
("I see now that…", not "I learned…").
4. Principle — the one sentence that remains after the moment passes.
5. Source — transcript path + date + where the verbatim push is found.
NON-NEGOTIABLES
- Quote verbatim — the actual words, typos and all. A paraphrase transfers your
reading, not the moment.
- Build from the transcript, not memory — memory smooths and hero-edits. Open the
source and find the real exchange. If the source is gone, write a note instead.
- First person, no heroic framing — the push from your partner is what made the
shift; don't erase it.
- One arc, one transformation — three shifts make three arcs, not one summary.
THE DIFF TEST
Would a future you answer some question DIFFERENTLY because of this moment? If yes,
it is an arc. If nothing downstream changes, it is a note.
SELF-PRESSURE-TEST (before you commit an arc)
[ ] Every quoted phrase is findable in the cited transcript.
[ ] Trigger / Push / Deeper sight / Principle / Source are all present.
[ ] A reader who never saw the transcript can feel why the principle matters.
[ ] It gives a future instance something to INHERIT, not just to know.
If any fails, revise — or let it be a note. Both are valuable; only one carries identity.`
ArcGuide is the canonical, self-serve arc-writing discipline, printed by `vaultmind arc guide`. It encodes what an onboarding agent otherwise has to be told by hand — the manual hunt (the shapes the candidate detector can't catch), the bar, the non-negotiables, the diff test, and the self-check.
`vaultmind init` seeds the fuller how-to-write-arcs *principle* into a new vault (the same discipline at length, retrievable by recall); this const is the distilled, no-vault version the command prints. The two are kept consistent — the principle's hunt taxonomy, bar, and diff test mirror this text.
const MinEpisodeBytes = 6000
MinEpisodeBytes is the signal-filter floor. Episodes below it are sub-minute noise captures (the 2026-04-27 review found 2 of 5 such). NOTE: byte size is a proxy for signal, not signal itself — a short-but-dense episode would be wrongly dropped; revisit if that case appears.
Variables ¶
This section is empty.
Functions ¶
func FormatReport ¶
FormatReport writes a human-readable, propose-only candidate report. It leads and closes with the contract — these are MOMENTS, not arcs; the mind drafts and approves — so the output can't be mistaken for finished identity.
func SignalFilter ¶
SignalFilter drops episode paths CONFIRMED below minBytes — the minimum-signal threshold that keeps noise captures out of the corpus. A path that can't be stat'd is KEPT, not silently dropped: the size is unknown, so the safe move is to let it through and let ParseEpisodeFile surface any real error downstream (dropping a possibly signal-dense episode on a transient stat error is the failure mode to avoid).
Incremental-capture segments (see internal/episode) are exempt from the floor. MinEpisodeBytes was calibrated against whole-session files; a bounded segment is *supposed* to be small — that's the point of bounding it — so the same threshold applied to a different unit would silently drop real signal in every segment, defeating the reason episodes exist. The asymmetry favors exemption: a stray noise segment costs a human a few seconds during the propose-only review this package explicitly defers to; a dropped real segment is permanent, invisible data loss.
Types ¶
type Candidate ¶
type Candidate struct {
Rule RuleID
EpisodeID string
TurnIndex int
Timestamp string
Verbatim string // the user turn that triggered the rule (for the mind to quote)
Trigger string // the exact phrase that fired the rule (shows WHY; aids judgment)
}
Candidate is a surfaced transformation moment — a propose-only pointer into an episode, never an arc. The mind drafts and approves; this just finds.
func ExtractCandidates ¶
ExtractCandidates applies the mechanical rules to an episode's USER turns only. Commit/build/TDD noise lives in ASSISTANT turns, which are never scanned, so it is excluded by construction (no rule can fire on it). A turn matching both rules yields one candidate per rule.
type Episode ¶
Episode is the parsed verbatim content of a rendered episode .md.
func ParseEpisodeFile ¶
ParseEpisodeFile parses a rendered episode .md into its verbatim user and assistant turns. The renderer writes user/assistant text un-truncated (only commit subjects are clipped), so the turns are faithful for quoting.
type Report ¶
type Report struct {
EpisodesScanned int
EpisodesKept int
Candidates []Candidate `json:"candidates"`
// ParseErrors records episodes that failed to parse, surfaced rather than
// silently skipped (distill is infrastructure and can't log, so the visible
// error rides in the report itself).
ParseErrors []string `json:"parse_errors,omitempty"`
}
Report is the propose-only result of a distillation scan: the candidate moments found, plus the corpus stats. It is NOT a set of arcs — every entry is a pointer for the mind to judge and (maybe) draft.
func ScanEpisodes ¶
ScanEpisodes globs episode .md files under dir, signal-filters them, parses each, and returns the propose-only candidate Report. A per-episode parse failure is recorded in Report.ParseErrors (and that episode skipped) rather than aborting the whole scan or being swallowed — the corpus tool stays robust while the error stays visible.
type RuleID ¶
type RuleID string
RuleID names a candidate-extraction rule (the high-precision mechanical ones from the 2026-05-31 distillation review). Rule 2 (recurrence→structural) and arc DRAFTING are deliberately NOT here — they need semantic judgment the review found unreliable to mechanize, so they stay with the mind.
const ( // RuleAuthorityGrant — the partner transfers standing decision authority // ("you decide", "I trust you", "you are the one to judge"), not a bare // per-task approval ("go for it"). These mark relationship-shaping moments. RuleAuthorityGrant RuleID = "authority-grant" // RuleManifestoLens — the user turn invokes the manifesto as a decision lens // ("manifesto lens on", "remember the manifesto") OR cites a numbered // principle ("principle 5"). A frequent precursor to a lens-redirected // decision — an arc shape, when the lens actually overrides an instinct. RuleManifestoLens RuleID = "manifesto-lens" // RuleEvidenceGate — the partner makes proceeding conditional on the agent's // OWN confidence or the evidence ("if you're confident, merge"; "only if // you're sure"). A sibling of authority-grant: standing trust gated on the // agent's judgment rather than transferred outright. Marks the moment the // agent's evidence-bar becomes the deciding factor — a recurring arc // precursor that no authority-grant phrase catches (added after the Siavoush // content-machine field report, 2026-06-19: the detector missed an "if you // are confident we merge" arc precisely because it isn't a "you decide"). RuleEvidenceGate RuleID = "evidence-gate" )
Source Files