Evaluating answers without fooling myself
Every time I changed something — a prompt, the chunk size, the model — I'd try a few questions and feel that the answers were better. Then a week later I'd notice something that used to work had quietly broken. Vibes are not a test suite.
The test set
I went through my shell history and my browser history for searches inside my own notes folder and picked 60 real questions. For each one I wrote down which note contains the answer and a one-line version of the answer itself. That took an evening and it's the most useful thing in the repo.
Two things I score separately:
- Retrieval: is the right note in the top five? Fully automatic.
- Answer: is the answer correct, and does it cite the right note? I grade this by hand, with the system output shuffled and the configuration name hidden, so I don't know which version I'm grading.
The “I don't know” questions
Ten of the 60 questions have no answer in my notes. A good system says so. The first version confidently answered seven of them with something plausible and wrong. After adding an explicit instruction and a minimum similarity threshold, it's down to one.
Current numbers
| v0.1 | v0.3 | |
|---|---|---|
| Retrieval top-5 | 63% | 88% |
| Correct answer with right citation | 51% | 79% |
| Made-up answers to unanswerable questions | 7 / 10 | 1 / 10 |
Grading 60 answers by hand takes about 40 minutes, so I only do it for changes I actually intend to keep.