NoSQL Schema Evolution and Big Data Migration at Scale II

NoSQL database systems for managing large volumes of data have become firmly established in software development. In a project funded by the German Research Foundation (DFG), a team at the University of Passau led by Prof. Dr.-Ing. Stefanie Scherzinger is researching tools for schema evolution and data migration.

The DFG project “NoSQL Schema Evolution and Scalable Big Data Migration” focuses on schema evolution in database applications that use NoSQL database management systems. Although many NoSQL DBMS do not define a fixed database schema, the application code nevertheless imposes certain requirements on the structure of the data; in other words, the application code assumes a particular schema. As the application code evolves, the schema must evolve along with it. The German Research Foundation (DFG) is funding the project, led by Prof. Dr.-Ing. Stefanie Scherzinger, Chair of Computer Science with a focus on Scalable Database Systems at the University of Passau, for a second funding phase.

During the first phase, the researchers focused on developing theoretically grounded tools for schema evolution and data migration in NoSQL databases. This research resulted in a number of well-received publications as well as, in particular, practical research prototypes.

In the second phase, the team aims to generalize this work to polystores, which combine multiple database management systems (DBMS) based on different data models. Although polystores are particularly well suited to continuously evolving agile applications that need to manage large volumes of data, research on schema evolution for polystores is still in its infancy:

• To date, there is no systematic approach to schema evolution for polystores.
• To date, there are no established best practices for the automated optimization of logical schema design. Moreover, such schema designs need to adapt to schema evolution.
• To date, there are no well-founded strategies for physical schema optimization in NoSQL databases that adapt to schema evolution.

In this DFG project, the team is investigating how these challenges can be addressed through a holistic approach spanning the global polystore schema, the logical schemas of the individual DBMS within the polystore, and their physical schemas.

During the second phase, the team is compiling an overview of the current state of the art, identifying open research challenges, and systematically developing solutions to them within the project. As a lasting contribution to the research community, the team is developing the first schema evolution benchmark for polystores and will make it publicly available to researchers.