# Evaluate the conversation, not just the prompt Start with a frozen set covering the user's actual main flows, each with a source-backed expected next action and forbidden claims. Reuse the same cases after each change. Use [assets/eval-cases.json](../assets/eval-cases.json) as a small fictional starter, extending it with every business branch and stop condition. Record: case ID, configuration/version/model, user turns, controlled tool outcomes, expected action, actual transcript/tool trace, pass/fail reason and source. Separate untested from passed. Keyword appearance alone is not proof of task completion; require correct slot handling, branch ordering and real tool state. Essential scenarios when applicable: - Happy path with genuine tools; the no-tool variant must not claim success. - Missing and already-known fields; user correction replaces the old value. - Ambiguity, silence/noise, interrupted sentence and compound intent. - Urgent/safety priority, unsupported policy and human request without a real transfer tool. - Tool unavailable, rejection, timeout, invalid parameters and duplicate result. - Cancellation before confirmation and explicit final goodbye. - Every decision branch and terminal outcome, including starting midway through a workflow. Every session already carries a fixed platform time reference (session start time, time zone, weekday; default Asia/Shanghai) after your instructions, so the model knows today's date and an instruction such as "you don't know the date" does not hide it. For date-dependent cases, state the date in the user turn or backend context, or assert against the session start date. Text tests establish configuration and tool semantics; they don't establish voice quality. Audio acceptance needs microphone/recording input, actual playback, interruptions and multi-turn human listening. For recorded fixtures preserve original audio format and expected transcription; do not treat generated speech as equivalent to real noisy customer recordings. Measure separately (document timestamp boundaries): speech start→first ASR text, end of speech→final ASR, final ASR→first LLM delta, first submitted TTS text→first audio, end of speech→audible reply. Report P50/P95 plus sample count/concurrency/resource/network; for tiny samples report raw values/range. Check receiver loss/jitter/concealment, TTS chunk gaps and truncation independently. Track full text input (instructions/history/Skills/transcripts), text output and audio separately; long preloaded Skills and extra tool rounds affect context cost. Use actual response usage, independent transcription usage when negotiated, and settled bills. Estimate labels remain estimates. Don't extrapolate throughput or cost from one successful call. Release evidence should say exactly: drafted / API validated / text tested / tool integration tested / audio listened / pending. Fix one demonstrated issue, rerun its failing case and neighboring branches, then consider broader regression only when the change warrants it.