confcal

Overview ¶

Command confcal calibrates the confidence gate. It collects the top-1 score distribution for queries whose answer EXISTS (gen corpus, split by whether the gold symbol was actually returned) and for queries whose answer does NOT exist (negprobe-style absent concepts), then reports, for every candidate threshold, what a gate at that threshold would cost and buy.

Why absolute score: retrieval_health gates on the RELATIVE top1-top2 gap only, so a query with no real answer still reads "high" whenever one junk candidate happens to lead. The cross-encoder emits a sigmoid, so its score is comparable across queries — an absolute floor is the missing signal.

Usage:

confcal -bin dist/contextmaxxer.exe -manifest internal/eval/testdata/manifest.public.json