PyTestGuard: An IDE-Integrated Tool for Supporting Developers with LLM-Generated Unit Tests

Nada Mouman will be defending her Masters’ Thesis on supporting developers in removing quality concerns from LLM-generated tests

Abstract

Unit testing is an important step in the software development workflow to detect bugs and ensure system correctness. Recently, Large Language Models (LLMs) have been explored to automate unit test generation and have demonstrated promising results. However, the generated tests are not always reliable, as they may contain syntax errors, hallucinations, test smells, or failing assertions. We conjecture that providing developers with feedback on such issues will increase the adoption of LLMs in real-world workflows. To address this, we propose PyTestGuard, a PyCharm plugin that allows developers to generate and refine unit tests directly within the Integrated Development Environment (IDE). Beyond test generation, PyTestGuard helps users evaluate test quality by detecting test smells and reporting issues such as missing arguments or references to non-existing objects. We conducted a user study with nine participants to assess PyTestGuard’s usefulness as a testing assistant and to identify areas for improvement. Participants reported that the tool’s feedback on test quality, along with its summarised error messages and coverage information, supported them while writing unit tests. However, they also faced challenges and suggested improvements before completely trusting LLM-based test generation in their development workflows. Based on these findings, we highlight several design recommendations for future tools that aim to integrate LLMs into software testing workflows.

Link to the MSc thesis on repository to be added