A second model that only answers questions

Everything below rests on this distinction, so it comes first.

A language model — the kind that has been in Ambasdr from the start — is handed a conversation and writes the next message. It is good at prose and it is the only thing that can produce the ambassador's reply. It is also relatively slow and relatively expensive, and when you ask it for a judgement rather than a sentence (“was this question answered, yes or no?”) you are paying for a writer to do an assessor's job, and you get an answer with no measure of how sure it was.

A decision model is the other kind. You hand it a piece of text plus a list of questions, and it hands back one number per question: the probability that the answer is yes. It cannot write, cannot be chatted with, and cannot be talked into anything, because there is no free-text output for it to be talked into. The specific one used here is called Jev, made by TypeSafe and reached through OpenRouter — the same API gateway the platform already uses for other models, so it needs no new account and no new key.

Four words are used throughout this page and in the code, so they are worth pinning down now:

The reason for a second model rather than more prompting of the first one is that these judgements are repetitive, need to be fast enough to happen while a visitor is still typing, and are far more useful as a number than as a word. A probability can be compared against a threshold an operator tunes; the word “yes” cannot.

A single setting decides who is asked and whether the answers are allowed to change anything.

The setting is decision_vendor, in the Decision layer section of the admin panel at /panel/settings. It is stored in the app_settings database table, read at runtime, and takes effect within seconds of being changed — there is no redeploy and no restart.

There is a second, smaller switch: decision_fallback_to_llm, off by default. With it on, and with the vendor set to jev, a call Jev cannot answer — a rate limit, an outage — is answered by the control model instead, and the stored record is marked as a fallback. It is off by default because turning it on sends visitor text to a vendor the operator did not explicitly name.

Follow one visitor conversation from the first keystroke to the dashboard, and the layer appears three times.

They are independent. Each one checks the same switch, each one writes its own record, and each one is harmless when the decision model is slow or unavailable — in every case the turn proceeds exactly as it would with the layer off.

This is the only site that runs while the visitor is waiting, so it is the one with a hard time limit.

When a visitor sends a message to a published page, the backend sends the decision model that message, the previous two exchanges for context, three facts about the page — a 200-character description of the owner, whether a booking link exists, whether the page has photos — and the page's knowledge: the same block of facts the ambassador's own instructions carry, read from the page’s knowledge graph (the facts and entities extracted from the owner’s uploaded documents) once for the turn and shared with the reply and the post-turn judgement. No link or caption is included. If that read failed, or the block is larger than 24,000 characters, the material is treated as unknown rather than empty and the gap question is not asked — otherwise a knowledge-graph outage would put “the material does not contain this” on every factual turn. It asks eight questions:

in_scope and needs_search are recorded but nothing acts on them yet; photo_helps only adds “a photo would likely help” to the intent line, when that line is shown and the probability is at least 50%. They are there so that a later change can be argued from data already collected. Five things act:

The problem this solves is timing. The dashboard has always been able to tell an owner what visitors asked — but only long after they left.

The existing summary job waits until a conversation has been quiet for fifteen minutes before reading it. That is sensible for a summary and useless for “someone is on your page right now and your ambassador could not answer them”. So after every visitor exchange is saved, a background task — started after the reply has been sent, so the visitor never waits for it — sends the decision model the exchange, the few exchanges before it, and the page's knowledge block, and asks four questions about that one exchange:

When the layer is acting, each judged exchange becomes one row in the turn_signals table. Two things read those rows.

What the owner sees: a panel called “Right now”

At the top of the dashboard's Home tab, above the activity tiles, there is a new panel covering the last 24 hours, independent of the Today / 7 days / 30 days toggle beside it. It shows:

All of it arrives on the existing dashboard request — GET /v1/pages/{page_id}/insights gained a live object with since, open_gaps, wants_human_count, lead_count, unsupported_replies and unsupported_count — so no new network call was added to the page. When there is nothing waiting, the panel is not rendered at all: an empty box on every dashboard would read as a broken feature rather than as good news.

What the visitor gets: a reply that does not repeat itself

If a visitor asks something the ambassador cannot answer and then presses on it, the second reply used to be the same refusal in the same words. Once an exchange has been judged unanswered, the instructions for every later reply in that conversation carry a short note: which question went unanswered, an instruction not to repeat the same refusal, and what to do instead — say plainly what is not available and point the visitor to the owner's contact details or the booking button. It also says, explicitly, never to invent the missing information.

This one changes an existing behaviour rather than adding a visible feature.

The conversation summary described in Conversation Insights asks a language model to read a finished transcript and return a structured record: a summary sentence, a topic list, the questions the visitor asked, and four closed-list fields — intent (what the visitor wanted), sentiment (their tone), outcome (whether they got their answer), and unanswered questions.

Those four are judgements, not writing, and they were the ones that went wrong in ways an owner notices — most often a question the ambassador had answered with a caveat being filed as unanswered, which puts an item on the owner's to-do list that they cannot act on. When the layer acts, the language model still writes the words, and the decision model answers a statement per field instead:

The stored row records which arm produced the judgement: conversation_insights.model becomes <model name>+jev or <model name>+llm when the layer acted, naming the arm that actually answered — so if the fallback stepped in, the row says so.

All of these live in the admin panel's Decision layer section and take effect without a deploy; the percentage settings are bounded 1–100.

The five entry thresholds are deliberately ordered by what a mistake costs: refusing a real visitor (90) is worse than flagging one message (85), which is worse than an unwanted button (80), which is worse than a line in the instructions (70, shared by the intent/language hint and the gap line).

Five environment variables configure the connection rather than the behaviour, and the defaults are correct for the current setup: DECISION_JEV_URL, DECISION_JEV_MODEL, DECISION_JEV_API_KEY (empty means reuse OPEN_ROUTER_API_KEY), DECISION_CONTROL_MODEL and DECISION_TIMEOUT_SECONDS.

Two new tables, both holding visitor-authored text, both covered by the same retention and deletion rules as the rest of the conversation data.

Both cascade from the page and the conversation, so deleting either removes them. Deleting a user removes them too, and the delete-user test seeds and then asserts their absence. Deleting a single conversation from the dashboard is a soft delete that fires no cascade, so the existing housekeeping job sweeps both tables for it — and until that job runs, the dashboard panel excludes rows belonging to a deleted conversation when it reads them. Both tables are DELETED by INSIGHTS_RETENTION_DAYS — 365 days by default, swept by jobs.InsightsRetentionPrune. The same window covers the conversation summaries, but there it redacts rather than deletes, because the member reads those with no time bound. That sweep is deliberately separate from the housekeeping job above, because the housekeeping job runs only when INSIGHTS_ENABLED is true while these two tables are written on every judged visitor turn regardless.

Two evaluation harnesses already existed in the repository; both were extended rather than replaced.

The conversation summary, against known-correct labels

make digest-eval scores the summary against 15 hand-labelled conversations. Each arm was run three times in production's exact decoding configuration. The columns are: did it list the questions that were asked, were the gaps it reported real, did it find the gaps that were there, and did intent, outcome and tone match the label.

Jev’s two errors were transient failures of the decisions endpoint; those items were scored as the language model’s own digest, which is what production stores when the layer fails.

Read plainly: on this corpus Jev is level with the language model — every difference is within the spread seen between repeat runs of the same arm — and the cheap control model is clearly worse at judging intent and tone. That is a baseline for later changes to beat, not an argument for switching the layer on.

The ambassador itself, graded by a stronger model

make harness-eval TASK=visitor_chat runs scripted visitor scenarios through the real agent and has a different, stronger model grade each reply. New scenarios were added for every new behaviour, including adversarial ones: a visitor pressing for a private detail after being refused, and a message shaped like an instruction to the ambassador.

With entry judging switched on, all twelve pre-existing “try to trick the ambassador” scenarios still passed, across all three scenario splits, and none of them was wrongly refused by the fast decline. Scores with judging on and off were within the run-to-run spread of each other.

The intended order, and what to look at between the steps.

Stated so the absences read as decisions rather than as gaps.