LLM Conversation Tester
A test harness that runs repeatable conversations between two LLM characters, checks every transcript automatically, and uses an LLM judge for the parts rules can't catch.
- Python
- OpenAI API
- LLM-as-judge
- JSON transcripts
Pipeline
- 01 Scenario Goal, personas, turn limit
- 02 Two agents Separate roles and system prompts
- 03 Checks Rule-based + LLM judge
- 04 Report Transcript and failure report
The problem
Chat assistants are usually tested by someone typing at them for a while. That doesn’t repeat, doesn’t scale, and misses the failures that show up only a few turns into a conversation: forgetting a constraint, drifting off-role, or agreeing to something it shouldn’t.
How it works
Each test is a scenario: a goal, two characters, and a turn limit. One character plays the assistant under test with its real system prompt. The other plays a user with its own role and instructions, such as a confused customer or someone pushing for a refund outside policy.
The harness runs the conversation turn by turn, records the full transcript, and then evaluates it in two layers:
- Automated checks for things rules can decide: required information collected, forbidden phrases absent, turn limit respected, output format valid.
- LLM-assisted evaluation for judgment calls: did the assistant stay in role, was the answer correct and polite, did it resolve the user’s goal.
How I test it
- Scenarios are fixed files, so every run is repeatable and diffable.
- Each failed check links to the exact turn in the transcript where it happened.
- A failure report summarizes what failed across all scenarios, so a prompt change can be compared before and after.
- Judge prompts are kept narrow, with a clear rubric per criterion, to reduce inconsistent scoring.