template_lab

template_lab

Microbench harness and sweep runner for template compression on extracted corpora.

It consumes corpus files generated by template_corpus_extract and reports:

• raw vs encoded bytes
• raw vs encoded gzip bytes
• encode/decode ns per raw byte
• attempted/matched/kept template counts
• rejection reason counters

Input format

Corpus files use repeated records encoded as:

• u32le length + raw payload bytes

Example

go run ./TreeDB/cmd/template_lab \
  -corpus-dir /tmp/template_corpus_fast \
  -dataset both \
  -outer-leaf-pretransform off \
  -disable-mask-templates false \
  -warmup-passes 1 \
  -measure-passes 1 \
  -sweep-min-savings 1,4,8 \
  -sweep-fingerprint-k 8 \
  -sweep-max-fetch 8,16 \
  -out-json /tmp/template_lab/results.json \
  -out-md /tmp/template_lab/results.md

Notes

• -include-off=true adds an off baseline row.
• -dataset both runs separate rows for pointer and outer_leaf corpora.
• -max-records caps records per dataset for fast local iteration.
• -outer-leaf-pretransform header_v1 enables a reversible outer-leaf header normalization prototype for lab experiments.
• -outer-leaf-pretransform header_dir_delta_v1 additionally delta-normalizes outer-leaf directory metadata (lossless) for experimental matching.
• -disable-mask-templates=false (default) enables full template mode. Set true for anchor-only diagnostic runs.
• Optional template training/routing overrides are available for investigation:
  • -template-train-sample-stride
  • -template-synthesize-every
  • -template-min-anchor-freq
  • -template-min-presence-ratio
  • -template-min-publish-savings
  • -template-min-publish-ratio
  • -template-cold-search-after
  • -template-cold-search-probe-every