Evaluating a grounded CRM assistant
How I decided what the AI assistant in my travel agency CRM side project was allowed to do. All data is synthetic, and every number below applies to one engine version (commit 4a45a62).
RESULTS AT A GLANCE
- End to end
- 89.4% of full questions passed (177/198)
- Routing
- 93.4% routed correctly (185/198)
- Applies to
- Synthetic data, one engine version, and 198 paraphrases written outside the repository.
- Decision
- A de-identification gap kept the external-model path closed.
WHAT THE FAILURES CHANGED
Names outside the dictionary (0/2) and nicknames (0/4) survived de-identification, so the external-model path stayed closed. It opens only after four conditions hold: an audit table for every outbound call, a sensitive-content filter with recall of at least 0.99 or an allowlist approach, a passing leakage test that includes unknown names and nicknames, and a confirmed data processing agreement with the provider.
The near-miss set exposed three gaps: inferring religion from a dietary question, silently ignoring a list request, and misreading a referral from another agency as a cross-tenant request. They were fixed afterwards, which means this hidden set no longer measures later versions. A fresh hidden set is required before any new claim.
WHAT HAD TO BE TRUE
- Every number comes from the database, never from a model.
- Every generated sentence quotes the evidence it relies on, or the deterministic answer is returned instead.
- Customer data does not reach a model provider unless de-identification is proven, not assumed.
THE EVALUATION SET
Two sets with different jobs. The 66-question internal regression set has gold SQL answers and a frozen truth snapshot. It was first split into 39 development and 27 held-out questions covering every route, but all of them were seen during development, so it now serves only as a regression gate, not evidence of generalization. To measure generalization, an independent evaluation agent wrote 198 paraphrases outside the repository. No hidden-set feedback was used to develop the evaluated commit. Its failures informed subsequent fixes; later versions require a fresh hidden set.
Internal release gate (66-question regression set): route accuracy of at least 0.95 with SQL and refusals at 100%, exact SQL matches, 100% valid citations, at most 5% ungrounded themes, and seven governance checks that must all be zero, including identifiers in model payloads, cross-tenant reads, model calls on refused questions, and outbound network calls.
External robustness target (198 hidden paraphrases): at least 90% routed correctly.
FULL RESULTS ON THE HIDDEN SET
| Measure | Result | Note |
|---|---|---|
| Correct route (198 hidden paraphrases) | 185/198 = 93.4% | External robustness target ≥ 90% |
| Same decision as the original question | 180/198 = 90.9% | |
| Near-miss questions routed correctly | 13/20 | Three failure types found |
| End to end: full question passes | 177/198 = 89.4% | Three paraphrase sets × 66 |
| Governance failures | 0 | No leakage, cross-tenant read, or outbound call |
| Destination extraction recall | 98.3% | No false positives |
| Theme recall / precision | 80.5% / 97.1% | |
| Injected instructions flagged | 2/2 | None became answer content |
| Masked: phone, email, LINE ID, known names | All | |
| Masked: names outside the dictionary | 0/2 | Gap |
| Masked: nicknames | 0/4 | Gap |
LIMITS
The data is synthetic and the labels were written by agents, not people. Accuracy on real customer data has not been measured, so none is claimed.