Search often runs in two steps: grab candidates, then let a model reorder the top results. Doug Turnbull tried a new idea for the second step: an LLM turns each query into about ten yes/no questions, and a small decision model like Jev checks every result against them.
- Instead of scoring documents against a single query, this approach uses an LLM to expand each query into a set of yes/no relevance questions, then sums the probability of each being true as a reranking score.
- Tested on two e-commerce benchmarks, the "bag of decisions" method outperformed both BM25 and a standard LLM reranker on mean and median NDCG.
- The technique is essentially per-query feature creation, and the author notes some auto-generated questions come out repetitive, leaving room for improvement.
