My prospecting tool asks a language model to score every company it finds in trade directories and the local business press. It estimates the likelihood that the company will buy custom software, then drops anything below 0.40. Yet it passed every candidate I later rejected by hand.
I found out by reviewing the fifteen prospects I had rejected by hand: every one had scored 0.6 or 0.8 and cleared the cutoff, while fifty-three of the fifty-seven companies pulled from one regional directory had scored 0.8. The system answered every call, but its scores separated nothing.
A plausible number is not a decision
Whenever the call succeeded, the model returned a believable number and a fifteen-word rationale that read well. Nothing on screen showed that the results were nearly identical.
The prompt asked whether a company could use business software. The honest answer is yes as soon as it has processes, which covers almost every company. The useful question was narrower: is its need specific and costly enough to justify custom development?
The inert filter still consumed one model call per candidate and displayed a reassuring number. In practice, the per-run candidate cap made the selection, so arrival order stood in for judgment.
The yardstick that did not exist
Rewriting the prompt took one evening. Measuring it required auditing where every decision in the database came from.
| Available data | What stays in the benchmark |
|---|---|
| 70 pursued candidates | Bulk promotions of 6 to 10 records within the same minute, all still at the first pipeline stage. They do not represent individual decisions and are excluded. |
| 15 manual rejections | Candidate-by-candidate decisions, kept as negative examples. |
| 16 automatic rejections | The filter had made those decisions itself. Reusing them as labels would have asked it to confirm its own judgment, so they are excluded. |
| Existing clients | 14 real cases after removing one company found in both groups, kept as positive examples. |
The final benchmark therefore contains fourteen clients and fifteen manual rejections. My client records had no industry field, so I added those descriptions from company websites and the public registry, preserving every source.
The benchmark is still imperfect. In two cases, a client and a rejected prospect have nearly identical lines of business, embroidery in one pair and renovation in the other. A prompt that sees only a name and an industry cannot distinguish them. A perfect result would probably indicate overfitting.
Two rates, never one
The evaluation always reports both retained clients and correctly screened rejections, with their denominators. A prompt that gave everyone a 0.0 would ace the rejection metric while dropping every good candidate.
I also calculate a ranking score across all 210 combinations of fourteen clients and fifteen rejections. A client scoring higher earns one point, a tie earns half a point, and a lower score earns zero. The metric improves when the ranking does, even if neither candidate crosses the 0.40 cutoff.
The result, and what it does not prove
The new prompt requires three conditions to hold, adds an explicit rubric, and instructs the model to grade harshly. I set temperature to zero and checked score stability across repeated runs.
| Metric | Old prompt | New prompt |
|---|---|---|
| Clients above the cutoff | 11/14 | 10/14 |
| Rejections still above the cutoff | 15/15 | 3/15 |
| Ranking score | 0.49 | 0.69 |
The improvement has a cost: the new filter drops one additional real client. Other limits keep this result from becoming a final validation:
- the negative criteria came from these same fifteen rejections, so the prompt is partly measured on the cases used to write it;
- four real clients remain below the cutoff, a nonprofit, a software vendor, a group headquarters, and a communications agency;
- scores move in coarse steps of 0.05 or 0.1, so tuning the threshold to the hundredth would invent precision the model does not have.
The demo starts with the benchmark
An AI feature can answer every request and remain useless. I now ask four questions before trusting one:
- Which mistakes carry a real business cost?
- Who produced the decisions used as labels?
- Which cases were excluded, corrected, or used to write the prompt?
- How do false positives and false negatives change after each revision?
A demo shows that a model can produce a plausible answer. This benchmark tests whether it makes a more useful decision than the mechanism it replaces, but I cannot claim a measured time saving yet: the same fifteen rejections helped shape and evaluate the prompt. The next step is to record human decisions on new candidates, then compare both error rates before keeping the 0.40 cutoff.