CSE1110 - Software Quality and Testing
Home • Teaching Team • Contact • Schedule • JPacman labwork • Grading • FAQ
This course covers the most important testing techniques needed to build high quality software systems. Specific topics covered are quality attributes, maintainability and testability, manual and exploratory testing, automated testing, devops, test adequacy, model-based testing, state-based testing, decision tables, reviews and inspections, design-by-contract, test-driven design, unit versus integration testing, mocks and stubs.
Our learning goals are:
-
Participants are able to create unit, integration, and system tests using current existing tools (i.e., JUnit, Mockito, and JaCoCo) that successfully test complex software systems.
-
Participants are able to derive test cases that deal with exceptional, corner, and bad weather cases by performing several different techniques (i.e., boundary analysis, state-based testing, decision tables).
-
Participants are able to measure and reflect about the efficiency of the developed test suites by means of different test adequacy metrics (i.e., line, branch, condition, MC/DC coverage).
-
Participants are able to reflect about limitations of current testing techniques, when and when not to apply them in a given context, and to design testable software systems.
-
Participants are able to write maintainable test code by avoiding well-known test code smells (e.g., Assertion Roulette, Slow or Obscure Tests).
A lot of the material of this course is also available in our EdX MOOCs on Automated Testing (Practical and Advanced skills for Java Developers). While these MOOCs are archived, if you’re interested you can still watch the videos and the exercises.
Teaching team
Head TA
Head TA
Contact
- Questions about the content: Brightspace forum, TAs during labwork, teachers during the break (no e-mail to the teachers).
- Infrastructure problems: contact Casper Boone or Max Lopes Cunha.
- TA availability: lab sessions. No direct Mattermost messages.
- Personal questions, mail Maurício and Arie.
Schedule
- Welcome to SQTE
- The JPacman Labwork
- Midterm and exam
- Revisiting the JUnit framework
- Introduction: (pdf)
- Labwork slides: (pdf)
- (Automated) software testing (automation): (pdf)
- Pragmatic software testing at SIGCSE 2019: (pdf)
- Find a team and register in Brightspace, Collaboration / Groups
- Read about our JPacman labwork and get started
- Read about the peer review system
- Read about our grading formula
- Read the most frequently asked questions
- The Oracle Problem in Software Testing: A Survey, by Barr et al.
- Chapters 1, 2, and 3 of our book
- Faults and failures
- Principles of testing
- Testing in the software development life cycle
- Validation and verification
- The V-Model
- Static analysis
- Foundations, ch. 1-3 (pdf)
- Matthew Yglesias. Boeing’s effort to get the 737 Max approved to fly again, explained". Vox.com, April 2019
- Dennis Muilenburg (CEO Boeing). Making steady progress on the 737 MAX software update. Boeing, April 2019.
- Phillip Johnston. What can software organizations learn from the boeing 737 saga? Embedded Artistry, April 2019.
- Phillip Johnston and Russ Ackoff. Beyond Coninual Improvement. Based on the presentation by Russ Ackoff. Embedded Artistry, February 2019.
- ACM Code of Ethics and Professional Conduct. ACM, 2018.
- Michaela Greiler. Code Review at Microsoft. 2019.
- Chapter 4 of our book
- Functional testing (testing from requirements)
- Boundary Testing
- On and off points
- Specification-based and boundary testing: (pdf)
- The category-partition method for specifying and generating functional tests, by Ostrand and Balcer
- A simplified domain-testing strategy, by Jeng and Weyuker
- Partioning, part I and part II
- Boundary testing, part I, part II, part III, and part IV
- The category partition method
- The CORRECT way
- The category partition method by our friends at Georgia Tech: video 1 and video 2
- Chapter 4 of our book
- Model-based testing
- Decision tables
- State-based testing
- Arie van Deursen. Beyond Page Objects: Testing Web Applications with State Objects. ACM Queue, 2015. (html).
- Decision Tables
- Model-based testing
- Decision tables
- Testing decisions
- Implementing decision tables
- Decision Table Wrap Up
- State Machines
- State machines
- Stacks as state machines
- State-based Testing
- Sneak path testing
- Super states
- Implementing state testing
- State-Based Testing Walk Through
- Interview with Lionel Briand on Model-Based Testing in Practice.
- Chapter 4 of our book
- Structural testing
- Line, branch, decision, condition coverage
- MC/DC coverage criteria
- Coverage subsumption
Jan Jaap Cannegieter is a leading test, QA and requirements expert from the Netherlands. He had over 25 years of experience in ICT. Jan Jaap did assignments in testing, test automation, quality assurance, process improvement, requirements, business – ICT alignment, Agile and digitalization. Jan Jaap is Principal Consultant at Squerist, a consultancy company of 100 employees specialized in process management and testing. Within Squerist Jan Jaap is, beside his assignments, responsible for coaching, knowledge management and product development. Jan Jaap is the well-known author of several articles and books in the Netherlands and is a well-known (keynote) speaker on international conferences.
Slides:- Exploratory testing: (pdf)
- What's testability?
- Design for testability
- Mock objects
- Unit vs system testing
- Integration testing
- The testing pyramid
- Mocks, part I, part II, and part III
- Testability
- Ports and adapters
- Dependency injection
- Practical tips on testability
- Our interview with Steve Freeman on mocks and testability
- The Hexagonal Architecture, by Alistair Cockburn
- Mocks aren't stubs, by Martin Fowler.
- The practical test pyramid, by Ham Vocke
- The Testability Guide, by Misko Hevery
- Classifications of security testing
- Java SAST tools and techniques
- SAST under the hood
- Security testing: (pdf)
- Dynamic Application Security Testing (DAST)
- Fuzzing
- Fuzzing for vulnerability hunting
- Security testing: (pdf)
- Buffer overflow example: Code snippet
- Short talk: How the solar team tests their car (Pieter Tolsma)
- Test-Driven Development (TDD)
- Advantages and disadvantages
- Pragmatic TDD
- Growing Object-Oriented Software Guided by Tests, by Freeman and Pryce. Best book on the topic.
- JavaScript unit testing
- UI component testing
- Web end-to-end testing
- Web testing: (pdf)
- Design by contract
- Pre-conditions, postconditions, and invariants
- The Liskov Substitution Principle and Testing Class Hierarchies
- Property-based testing
- Selected questions from the mid-term exam
- Self-Testing: (pdf)
- John Regehr: Use of Assertions
- Search-based software testing
- Genetic algorithms
- Fitness function
- Automated test case generation
- Slides: (pdf)
Tim van der Lippe is a software engineer at Google in London. He is a former student of Delft University of Technology and former teaching assistant of this course. During his study and in his current job, he is the maintainer of two open source projects: web components library Polymer and Java mocking framework Mockito. In this guest lecture, he will give an overview of his daily work at Google, the problems of working at scale and what solutions were necessary to migrate Google to the latest version of Mockito.
Additional material:- Our interview with Tim on pragmatic software testing.
Grading
Your final grade is calculated as follows:
Grade = (L + 2 * max(M, E) + 2 * E) / 5
where:
- L is the labwork (L being the average grade of the three parts of the labwork) + peer review.
- M is the midterm
- E is the exam
The midterm and the exam both contain 40 multiple-choice question, with 4 options each. Their final grade is calculated as:
Grade = (C - W/3) / 4
where:
C is the number of correct answers and W is the number of wrong answers. You are allowed to not answer a question; in this case, it won’t count as a wrong answer, and thus, not part of the guessing adjustment. With such a formula, we suggest you to not answer a question if you have no knowledge of the question; on the other hand, if you have partial knowledge (i.e., you can discard 1, 2, or 3 choices out of the given 4), it is therefore best to guess.
Note that you are required to have L >= 5.75 and E >= 5.75 to pass. If you do not pass both the labwork and exam, your final grade is calculated as min(E, 4.0).
FAQ
-
Will there be collegerama recordings of this course? In the schedule above, we provide you with summary videos for all our lectures. For that reason we will not make TU Delft collegerama recordings.
-
Are the deadlines strict? Yes, deadlines are strict. If you miss the deadline, you get a 0. If you had an issue that prevented you from delivering, please talk to your student counselor.
-
Can I skip the midterm? Yes, you can. The midterm is not compulsory. However, note that the midterm can count as 40% of your final grade. You have less material to study for the midterm too. So, our suggestion is for you to study hard and take the midterm!
-
My friend is from group Beta, and I’m from group Alpha. Can we still be a team? Yes, you can. Keep in mind that your lab times will still be different. In moments where the lab is busy, TAs will prioritize students from that lab; on the other hand, in times that the lab isn’t too busy, you can work together there.
-
I did the JPacman assignment last year. Can I reuse the grade? If you did the JPacman assignment in 2018 and you got a passing grade, you do not need to do it again. Go to Brightspace -> Assignments -> Reuse 2018 JPacman grade, and reply it with a “Yes”.
-
Can I use my devices in class? Educational research clearly shows that using devices on non-class related activities can harm your learning. Moreover, it can also harm your colleague’s learning. Therefore, you are not allowed to use any devices during the lecture. Exception: activities that require devices. I will help you by making announcements during the lecture about the times you should use your device.
-
How did you do this website? This website is done with Jekyll and some straightforward HTML. The main picture was taken by Agence Olloweb.
-
Can I use your material?
This website as well as all the content created by me are licensed through Attribution-NonCommercial-ShareAlike CC BY-NC-SA. This license lets others remix, tweak, and build upon your work non-commercially, as long as they credit you and license their new creations under the identical terms.
The content from others that I link are subject the own authors’ licenses.