AI QA Reporting Agent
An agent that turns raw test results and defect data into a daily engineering report a lead can read in two minutes.
- Python
- TypeScript
- LLM tool calling
- Structured JSON output
Pipeline
- 01 Sources Test runs and defect list
- 02 Agent Tool calls gather and group
- 03 Structured draft Validated JSON
- 04 Daily report Summary, risks, open defects
The problem
QA status lives in several places: CI results, the defect tracker, and people’s notes. Putting it together for a daily update is repetitive work, and the version that gets written at 5 pm is often incomplete.
How it works
The agent pulls the day’s test results and defect data through tools, groups them by feature area, and drafts the report as structured JSON first: pass/fail counts, new and reopened defects, blockers, and risks. That JSON is validated against a schema and only then rendered into a readable report.
Separating the structured draft from the final text keeps the numbers exact. The model writes the explanations; it never invents the counts.
How I test it
- Schema validation on every draft, with a retry when the output doesn’t match.
- Counts in the report are checked against the source data before sending.
- Guardrails stop the agent from reporting a defect that doesn’t exist in the tracker.
- Traces for each run are kept, so a wrong statement can be followed back to the tool call that caused it.