Release Engineering for Machine Learning
📣 Announcement: 📣 The link to the the Zoom meeting was shared via Brightspace.
In this course, we will go on a journey that starts at continuous integration and then moves on to continuous delivery, continuous deployment, and continuous experimentation. We will discuss the theory and the current research on various related subjects like containerization, testing, or monitoring and will put the learned theory into practice. As a running example, we will build a pipeline for a machine learning application, which – compared to traditional release engineering – poses additional challenges, like data versioning or model deployment.
Learning Objectives
After following this course, you will be able to:
- Apply standard techniques of release engineering;
- Apply version control techniques to machine learning artifacts, like data or models Design a deployment pipeline for a machine learning application;
- Implement quality control techniques in a machine learning pipeline;
- Analyze and improve existing deployment pipelines;
- Evaluate and document design decisions in deployment pipelines.
Organization
| Course code | CS4295 |
| Instructors | Sebastian Proksch, LuÃs Cruz |
| Schedule | Mondays, 15:45–17:45 Wednesdays, 13:45–15:45 Fridays, 13:45–15:45 |
| ECTS | 5.0 |
| Quarter | Q4 |
| Examination type | Software project (35%); Essay (60%); Presentation (5%) |
| Target audience | Students of the M.Sc. programme in Computer Science |
| Requirements | - Intermediate understanding of OOP languages; - Practical experience with continuous integration; - Basic understanding of software testing principles; - Basic knowledge of machine learning techniques. |
Outline
Please note: The program below is tentative and is subject to change.
| Day | Week | Type | Summary |
|---|---|---|---|
| 19Â Apr | 1 | Lecture | Introduction: Organization and Course Schedule (slides); Tutorials and Final Project (slides). |
| 21 Apr | 1 | Lecture | Continuous {Integration, Delivery, Deployment} (slides, video). |
| 23 Apr | 1 | Lab | Deployment in Practice: GitLab and Digital Ocean (repositories, video). |
| 26 Apr | 2 | Lecture | Containers & Orchestration (slides, video). |
| 28 Apr | 2 | Lab | Containerization in Practice: Docker & Docker Compose (repositories, video (part 1), video (part 2)) |
| 30 Apr | 2 | Lecture | ML Testing. Recommended reading: What’s your ML test score? A rubric for ML production systems. [1]; also covering [2] and [3] (slides, video). |
| 3 May | 3 | Lecture | Part I – Guest Lecture: Alex Serban on ML Best Practices. Recommended reading: Adoption and effects of software engineering best practices in machine learning [6]. Part II – ML pipelines. (slides (part I), slides (part II), video) |
| 5 May | 3 | Free | Public Holiday |
| 7 May | 3 | Lab | Tutorial – ML Configuration Management (video). |
| 10 May | 4 | Lecture | Continuous Experimentation. (slides, video) |
| 12 May | 4 | Lab | Kubernetes and Monitoring (video) |
| 14 May | 4 | Free | Public Holiday |
| 17 May | 5 | Self-Study | Literature Survey |
| 19 May | 5 | Lecture | How to write a paper (slides). Presentation tips (slides). (video) |
| 21 May | 5 | Feedback | Review Current Pipeline, Pipeline Extension Proposal |
| 28 May | 6 | Feedback | First Draft of ToC + Intro |
| 4 Jun | 7 | Feedback | Individual Steering Meetings |
| 11 Jun | 8 | Feedback | Sketch of Methodology or new pipeline |
| TBD | 9 | ?? | TBD |
| 21 Jun | 10 | Examination | Essay and Presentation |
| ………… |  |  |  |
Recommended Reading
- [1] - Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley (2016). What’s your ML test score? A rubric for ML production systems. Reliable Machine Learning in the Wild - NIPS 2016 Workshop (2016) Preprint.
- [2] - Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2020). Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering. Preprint.
- [3] - Sun, Z., Zhang, J. M., Harman, M., Papadakis, M., & Zhang, L. (2020, June). Automatic testing and improvement of machine translation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 974-985). Preprint.
- [4] - Haakman, M., Cruz, L., Huijgens, H., & van Deursen, A. (2020). AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech. Preprint.
- [5] - van Oort, B., Cruz, L., Aniche, M., & van Deursen, A. (2021). The Prevalence of Code Smells in Machine Learning projects. Preprint.
- [6] - Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020, October). Adoption and effects of software engineering best practices in machine learning. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). Preprint.