SERG Seminar

In this edition of our weekly SERG seminar, we will hear from:

Michael Pradel on “Natural Language as Specification: LLM-Based Validation of Software Evolution”

Abstract:

Software evolves rapidly, yet determining whether a change behaves as intended remains challenging. Traditional regression testing reports any behavioral difference, while developer intent – expressed in pull-request titles, descriptions, and commit messages – remains informal and non-executable. In this talk, we present a line of work that treats natural language as a practical specification and operationalizes it using large language models (LLMs). We first introduce Testora, which generates tests for modified code and classifies behavioral differences as intended or unintended by comparing them against stated intent. We then present PatchGuru, which infers executable “patch oracles” from natural-language artifacts to enable automated cross-version checks and bug reporting. Finally, we present RippleGUItester, which shows that LLM-based change validation can also be applied to entire applications at the GUI level. Evaluations on real-world open-source projects show that these techniques uncover previously unknown bugs at reasonable cost with practical turnaround times, suggesting that development artifacts themselves can serve as specifications and that natural language can become an automated validation signal when paired with modern LLMs.

Bio:

Michael Pradel is a faculty member at the CISPA Helmholtz Center for Information Security and a full professor at the University of Stuttgart, which he joined after a PhD at ETH Zurich, a post-doc at UC Berkeley, an assistant professorship at TU Darmstadt. He has visited Facebook, UC Berkeley, and UCLA for sabbaticals. His research interests span software engineering, programming languages, security, and machine learning, with a focus on tools and techniques for building reliable, efficient, and secure software. Michael has been recognized through the Ernst-Denert Software Engineering Award, an Emmy Noether grant by the German Research Foundation (DFG), two ERC grants, best/distinguished paper awards at FSE (3x), ISSTA, ASE (2x), ASPLOS, and MSR, and by being named an ACM Distinguished Member.

If you are interested to give a talk or host a discussion session in one of our next meetings, please contact Carolin Brandt via Mattermost or email.