The research at our chair is centered around understanding the quality of computer systems, focusing on software quality, including the quality of systems that use Artificial Intelligence (AI). We are concerned with internal quality (e.g., maintainability), external quality (e.g., defects), and non-functional aspects like compliance.
Consequently, our research methods are diverse but can be grouped into three main categories: 1) understanding software engineering through repository mining; 2) understanding and improving the quality of machine learning models; 3) applying machine learning models.
Software repositories (e.g., GitHub) contain vast amounts of data about software engineering, which we can collect and analyze. We use this data to understand how projects evolve and derive general knowledge about software engineering, often focusing on bugs: how are they introduced, how are they reported, and how they are fixed. Repositories also allow us to study many other aspects, e.g., how tools impact code quality, which kinds of tests are used within projects, or the development practices employed, and how developers deal with non-functional aspects like data protection, ethical concerns, or EU AI Act compliance.
The rise of machine learning from a research topic to an enabling technology of innovative products means that we need to view machine learning software differently: instead of a tool for researchers for studying phenomena, modern AI powered by ML is now widely used for diverse tasks, such as image classification for quality gates, software engineering, drafting and analysis of texts. Moreover, while training models for specific purposes dominated until a couple of years ago, the use of foundational models without task-specific training is now a prominent usage paradigm.
Within our current work, we focus on Large Language Models (LLMs). We investigate their general capabilities (e.g., writing quality, adopting specific stances), to what degree foundational models generalize to specific domains, and how we can better adapt them to new domains, as well as how they work internally. Our goal is as important as it is ambitious: develop techniques to validate capabilities (e.g., types of reasoning, encoding of language aspects), identify the parts of the neural networks that encode these capabilities, and validate their causal responsibility for this functionality. This will pave the way for the segmentation of large neural networks from general purpose into smaller networks with specific capabilities, as well as raise our understanding of how such technologies work.
We believe quality cannot be studied in benchmarks but needs to be considered with respect to AI applications as well. Therefore, it is only natural to use our position in the intersection between software engineering and AI to also collaborate with end users and domain experts to develop and understand concrete AI applications in a research-oriented manner. Our collaborators provide domain knowledge, and we provide AI and tool-building expertise. We also study existing tools to understand them, e.g., to assess their compliance. Such collaborations are crucial for us, as they help us to shape our work on machine learning quality: we learn firsthand what matters to others, what they expect from machine learning tools, and can derive suitable research goals as well as evaluate the practical utility of AI solutions.
