Researchers present Foyer, a self-hosted personal inbox agent, alongside a rigorous measurement protocol designed to assess the real-world reliability of email triage systems. The starting problem is methodological: most inbox agents generate their own training labels, which biases any subsequent performance evaluation. The proposed protocol relies on stratified sampling, frozen snapshots, thread-grouped data splits and pre-registered hypotheses, with ground-truth labels provided by the inbox owner within the agent's own review loop.
Across 739 real emails in Italian and English, labelled by their recipient, nine systems are compared: production keyword rules, a TF-IDF plus logistic regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 at 26B and 31B parameters). One striking early finding: the production rules, despite being the most visible field in the interface, scored only 28% accuracy on message type against a 58% majority-class baseline — meaning the most prominent signal carried essentially no information.
With roughly 200 labelled examples, logistic regression achieved the highest accuracy on every field tested — message type, urgency, need for a reply, and noise detection — and was statistically tied with Gemma 4 on type classification. Few-shot prompting did not close this gap for the LLMs. Another notable result concerns inferring the sender's relationship (client, supplier, service provider): this was recovered with 98.6% accuracy when the contact had previously appeared elsewhere, but no model could infer it from a single isolated message. The authors conclude this signal belongs in a contact graph rather than in the prompt itself.
The most practically relevant finding concerns the gap between raw accuracy and real product value. Once predictions are translated into downstream actions, the most accurate system turns out to ignore emails that actually required action in 18% of cases, compared with just 4% for the best-performing LLM. Tuning a cost-aware decision threshold on the noise probability cut that failure rate in half while also making the system the cheapest to run. The study highlights a point often overlooked in AI agent evaluation: a high overall accuracy score can conceal critical failures on exactly the cases that matter most to end users.