ftdata

Overview ¶

Command ftdata turns CORE-Bench-style BEIR data into embedder fine-tuning triplets: {"query", "pos": [...], "neg": [...]} JSONL, one line per query.

Methodology guardrails baked in:

• The train/holdout split is BY REPOSITORY, decided deterministically (fnv1a(repoKey) % 100 < holdout-pct) plus a forced holdout list — so the model is never trained on any repo it will be evaluated on, and re-running ftdata reproduces the exact same split. The split is written to split.json next to the data; commit that file with any training run.
• Hard negatives are mined per query INSIDE the query's temporal filter (filtered_corpus_id): top BM25-ranked chunks that are not relevant. BM25 mining needs no GPU/model, so it can run while a benchmark owns the GPU; vector mining can be added later from the embedding caches.