Hermès, my automation tool, reads the messages I send it on Telegram. When I correct it, with a complaint, a standing instruction or a pointer to its mistake, a weekly agent turns the correction into a rule. To spot them, a regex catches the obvious phrasings and Claude Haiku reads whatever the regex lets through. I had compared Haiku with the regex, never with a trained model. I wanted to check whether a classic model trained on my own messages would do as well, faster and for free. The code and method are public on GitHub, without any of my messages.
Detectors and labels
I scored these detectors on the same 260 messages:
- the production regex;
- Haiku, called the way production calls it;
- the current pipeline, regex first and then Haiku;
- a logistic regression on TF-IDF;
- CamemBERT, a 110-million-parameter French language model, fine-tuned with PyTorch.
Claude Opus labeled all 260 messages without seeing what the other detectors said, then I settled 80 cases myself on Telegram: the disagreements, the labels Opus was unsure about, and 20 messages drawn at random. The two trained models go through 5-fold cross-validation, so they never see the message they are scoring, and every score comes with a 95% confidence interval.
The results
With 92 corrections in the set, two close scores can’t be told apart by eye, so I also compared the detectors pairwise on the same messages. On this sample, Haiku is clearly ahead of the regex (0.45 F1) and of CamemBERT (0.13). Against TF-IDF, the 0.09 gap is real but its interval comes close to zero, so I wouldn’t bet on its size.
What these numbers can tell
Not much beyond my own case. 260 messages from one person, in French, on a single task: the intervals are wide, a gap of a few hundredths proves nothing, and nothing says the ranking would hold on other data. The trained models only saw about 200 examples each, which favors the LLM. These results describe what you get with few labels, not the ceiling of each approach.
For Hermès, they point in a direction:
- keeping Haiku makes sense. At my volume, 260 messages in five months, it costs a few cents a year at API pricing, and it recognizes criticism of tone even when it isn’t phrased as a stock formula, while the regex catches only a fifth of the corrections;
- the regex in front of Haiku looks redundant: Haiku finds 19 of its 20 corrections, and the regex added one correction for three false alarms. That’s an argument for removing it, to be confirmed on the coming months of messages;
- CamemBERT does no better than TF-IDF here, which is no surprise with 200 examples for 110 million parameters.
When the choice would be different
TF-IDF ends up 0.09 behind Haiku and answers in a millisecond, for free, without sending anything off the machine. With a few hundred more labels, millions of messages a day, or data that must not leave the premises, the balance would probably tip its way. The same benchmark, at another volume or under another constraint, could reach a different conclusion, which is why it’s worth running rather than assuming.
What the measurement doesn’t see
Almost half of the corrections Haiku misses only exist through the conversation:
- a two-word follow-up after a failure;
- a pointer to a screenshot sent earlier;
- a reminder that something was already done.
No detector that reads one message and the end of the previous reply can see them. Giving it more context looks more promising than switching models, but I haven’t measured that yet.
The other limit is me. Among the 20 random messages where Opus, the regex and Haiku all agreed, I overruled Opus four times, and some of my own rulings contradict each other. Above 0.8, this benchmark probably measures agreement between annotators as much as the quality of the detectors.
If you’re considering an LLM to sort, classify or detect something, we can compare on your data what it brings over simpler options. It doesn’t always win.