IN4315 - Software Architecture

Describe decision-making process of pull requests

What to look for in a pull request?

Inspect each individual comment in each pull request to explain the decision-making process. Particularly, look for answers to the following questions: (a) why is a pull request accepted? (b) why is a pull request rejected? As you inspect each comment, look for factors that can potentially influence the decision-making. The outcome of analyzing each pull request should be your theory explaining (a) how a decision was reached? and (b) What factors influenced the process?

Please note that each pull request is unique, what works for one pull request may not work for another. Also, there is no right or wrong answer to this exercise. So, report what you think is most likely.

How to analyze a pull request and report findings?

For each pull request, report two things: (1) what is the context of a pull request? and (2) Why did developers reach the given decision? Below are some guidelines for each step:

To add context, provide information like: at which moment in the project lifetime was the pull request made? What components does it touch? Was it discussed as an issue before? Does it deprecate another pull request?

To codify the process, for each comment, define (or reuse) a code (think of it as a tag) that best describes the content. Next, analyze patterns in discussions using codes, organize the comments, if needed merge the comments (based on codes) to identify broader themes. You may need to look beyond the discussions to better understand the process.

The result of this exercise should be a theory or a set of alternate theories explaining why developers reached the given decision. The resulting theory is what you think might have happened based on the evidences gathered in the coding process.

To make the entire process robust, we encourage at least two team members to codify a pull request independently. Once the two team members have independently created a theory(-ies), the two results must be compared for interrater agreement. If the results match, the group has reached a theory explaining the decision-making of pull request. If the results do not match, the team members discuss the pull request unless they reach a common ground. Also, report the interrater agreement along with the theories before and after the discussion.