ae elsabban.dev
← All projects
AI agents In development

MCP Playwright Triage Server

An MCP server that hands Playwright test results to an LLM, so failed runs come back sorted into likely root causes instead of a wall of stack traces.

Context
Tech Horizon
My role
Design, build, and evaluation
Code
Private, walkthrough on request
  • TypeScript
  • Model Context Protocol
  • Playwright
  • Claude
  • JSON reporter

Pipeline

  1. 01 Playwright run JSON report, traces, screenshots
  2. 02 MCP server Exposes results as tools
  3. 03 LLM agent Reads failures, asks for detail
  4. 04 Triage report Root cause + suggested fix

The problem

A nightly regression run fails 12 tests. Someone on the team spends the first hour of the morning opening traces one by one to decide which failures are real product bugs, which are flaky waits, and which are test-data problems. The information needed to make that call is already in the Playwright report. It just isn’t organized for a decision.

How it works

The server speaks the Model Context Protocol, so any MCP-capable client (Claude Desktop, Claude Code, or a custom agent) can connect to it. It reads Playwright’s JSON reporter output and exposes it as a small set of tools:

  • List failures for a run, with test title, file, project, and retry count
  • Get failure detail: error message, stack, the failing step, and attached artifacts
  • Compare with history to see whether a test failed the same way in earlier runs

The agent calls these tools as it needs them instead of receiving one huge prompt. It then groups failures by likely cause (product defect, flaky timing, selector drift, environment or data) and writes a short triage note for each group with the evidence it used.

How I test it

An AI triage tool is only useful if its calls are right, so it gets the same treatment as any system under test:

  • Seeded failures. A practice suite with known, deliberately injected failures of each type gives a labeled set to score the agent against.
  • Structured output checks. The triage report is JSON with a fixed schema, validated on every run before anything is shown to a person.
  • Evidence requirement. Each classification has to cite the error text or step it relied on. Unsupported claims fail the check.
  • Regression on prompts. Prompt or tool changes are re-run against the labeled set so accuracy can’t silently drop.

What’s next

Posting the triage summary to the pull request that triggered the run, and adding trace-level detail (network and console) to the tools.