starlee
/
duppr_analysis
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

some refinement

This commit is contained in:
whystar 2020-06-25 19:36:16 +08:00
parent 97c637c810
commit 82bd4eabf0
20 changed files with 707 additions and 575 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
1_introduction.tex
View File

@ -10,7 +10,7 @@ Consequently, an increasing number of OSS projects are adopting the synthesized
which helps them improve their productivity~\cite{vas15qua} and attract more contributors~\cite{yu16det}.
However,
while the increased number of contributors in large-scale software development leads to more innovations (\eg unique idea and inspiring solutions),
while the increased number of contributors in large-scale software development leads to more innovations (\eg unique ideas and inspiring solutions),
it also results in severe coordination challenges~\cite{west2006cg}.
Currently,
one of the typical coordination problems in pull-based development
@ -29,13 +29,14 @@ it is impractical to expect external developers
(especially newcomers and occasional contributors)
to deeply understand the development progress of the OSS projects~\cite{lee2017understanding,steinmacher2015social,gousios2016work}
before submitting patches.
Thus, OSS developers involved in the pull-based model sometimes
submit \textit{duplicate pull requests} (akin to duplicate bug reports~\cite{bg2008dp}),
even though they collaborate on modern social coding platforms (\eg GitHub)
Thus, OSS developers involved in the pull-based model submit \textit{duplicate pull requests} (akin to duplicate bug reports~\cite{bg2008dp}), even though they collaborate on modern social coding platforms (\eg GitHub)
with relatively transparent~\cite{Tsa14Let, Dabbish2012Social} and
centralized~\cite{Gousios:2014} working environments.
{\color{hltext}
The recent study by Zhou \etal~\cite{zhou2019fork} has showed that full or partial duplication is pervasive in OSS projects and particularly severe in some large projects (max 51\% of duplicates).
}
Notably, most duplicates are not submitted intentionally
Notably, a large part of duplicates are not submitted intentionally
to provide different or better solutions.
Instead, contributors submit duplicates unintentionally
because of misinformation and unawareness of a project's status~\cite{gousios2016work,zhou2019fork}.
@ -53,12 +54,17 @@ assembled with DevOps tools compared to traditional development models.
For example,
continuous integration tools (Travis-CI~\cite{vas15qua, widder2019conceptual}) automatically merge every newly received pull request into a testing branch, build the project and run existing test suites,
so computing resources are wasted if integrators do not discover the duplicates and stop the automation process in time.
%Therefore,
%%%%% the collaborative environment, process and human factors that lead to duplicates should be clearly investigated and,
%the underlying factors leading to duplicate pull requests should be clearly investigated and,
%%%%% fixed %%%therefore后面也是come from nowhere. 你前面讨论的内容哪些是"collaborative environment", 哪些是process，哪些是human factors?
%if necessary, an automated strategy should be designed to
%detect and dispose of duplicates in a timely and efficient manner.
{\color{hltext}
Therefore,
%both project integrators and contributors hope
avoiding duplicate pull requests is becoming a realistic demand for OSS management, \eg
scikit-learn provides a special note in the contributing guide
\textit{``To avoid duplicating work, it is highly advised that you search through the issue tracker and the PR list. If in doubt about duplicated work, or if you want to work on a non-trivial feature, it’s recommended to first open an issue in the issue tracker to get some feedbacks from core developers.''}~\cite{intor-avoid}
%and \textit{``Oops! Sorry, did not mean to double up''}~\cite{cntor-avoid}.
}
Existing work has highlighted the problems of duplicate pull requests~\cite{Gousios:2014,steinmacher2018almost,zhou2019fork} (\eg inefficiency and redundant development), and
proposed ways to detect duplicates~\cite{li2017detecting,ren2019identifying}.

26
2_background.tex
View File

@ -110,7 +110,7 @@ or use qualitative methods to explore the factors that integrators examine when
\noindent \textbf{Characteristics of accepted contributions.}
% Prior research has extensively quantified the impact of various aspects of OSS contributions
% that affect their acceptance.
Rigby \etal~\cite{Rigby08apache} analyzed the patches submitted to the Apache server project
Rigby \etal~\cite{Rigby08apache} analyzed the patches submitted to the Apache server project,
and found that patches of small size are more likely to be accepted than large ones in that project.
The similar finding was also reported by Weissgerber \etal~~\cite{weissgerber2008small}
in their study on two other OSS projects.
@ -125,14 +125,14 @@ In the pull request model specifically,
Gousios \etal~\cite{Gousios:2014} found that the hotness of project area is the dominating factor affecting pull request acceptance.
Tsay \etal~\cite{Tsay2014Influence} investigated the effects of
both technical and social factors on pull request acceptance.
Their findings showed that the social connections
Their findings showed that stronger social connections
between the pull request author and integrators
can increase the likelihood of pull request acceptance.
Yu \etal~\cite{yu16det} investigated pull request evaluation in the context of CI (continuous integration).
They found that CI testing results significantly influence the outcome of pull request review and
the pull requests failing CI tests have a high likelihood of rejection.
Kononenko \etal~\cite{Kononenko2018Studying} studied the pull requests submitted to a successful commercial project.
Their analysis results presented that patch size, discussion length, and author experience and affiliation are important factors affecting pull request acceptance in that project.
Their analysis results presented that patch size, discussion length, and authors' experience and affiliation are important factors affecting pull request acceptance in that project.
A recent study conducted by Zou \etal~\cite{zou2019how} showed that
pull requests with larger code style inconsistency are more likely to be rejected.
Interestingly,
@ -157,7 +157,7 @@ Although technical issue is the dominating reason,
the reported reasons also include project scope and other political issues.
Pham \etal~\cite{Pham2013Creating} interviewed with project owners from GitHub,
and found that many factors are considered by project owners when evaluating contributions,
including the trustworthiness of contributor and the size, type and target of changes.
\eg the trustworthiness of contributor and the size, type and target of changes.
Tsay \etal~\cite{Tsa14Let} analyzed integrators' comments and decisions on highly discussed pull requests.
They found that integrators are usually polite to new contributors for social encouragement,
and integrators' decisions can be affected by community support.
@ -193,7 +193,7 @@ which makes it easier to submit bug reports.
Consequently,
popular OSS projects can receive hundreds of bug reports from the community every day~\cite{sun2011towards}.
However,
because reporting process is uncoordinated,
because the reporting process is uncoordinated,
some bugs might be reported multiple times by different developers.
For example, duplicate issues account for 36\% and 24\% of all reported issues in Gnome and Mozilla~\cite{zhou2012make}, respectively,
and consume considerable effort from developers to confirm.
@ -242,18 +242,18 @@ in both the form and the process.
\vspace{0.5em}
\noindent \textbf{Duplicate pull requests.}
{\color{hltext}
% {\color{hltext}
Figure~\ref{fig:dup_example} shows an example of a pair of duplicate pull requests,
both of which replace function \texttt{empty?} with \texttt{any?} for better readability.
Duplicate pull requests both go through the normal evaluation process
until their duplicate relation is identified by the reviewers.
In addition to peer code review,
various kinds of DevOps tools~\cite{hil16usa,vas15qua,beller2016analyzing}
have been commonly used to check the quality of pull requests.
Each new submitted pull request and any subsequent commit pushed to a pull request automatically trigger the DevOps tools.
Consequently,
duplicate pull requests also consume unnecessary DevOps computing resources if the reviewers do not detect the duplicates in a timely manner and manually stop the DevOps tools.
}
% In addition to peer code review,
% various kinds of DevOps tools~\cite{hil16usa,vas15qua,beller2016analyzing}
% have been commonly used to check the quality of pull requests.
% Each new submitted pull request and any subsequent commit pushed to a pull request automatically trigger the DevOps tools.
% Consequently,
% duplicate pull requests also consume unnecessary DevOps computing resources if the reviewers do not detect the duplicates in a timely manner and manually stop the DevOps tools.
% }
Prior studies have reported that duplicate pull requests are a pervasive and severe problem that affects development efficiency~\cite{steinmacher2018almost,zhou2019fork}.
Gousios \etal~\cite{Gousios:2014} manually analyzed hundreds of unmerged pull requests. They found that more than half of them were rejected due to non-technical reasons and that duplication is one of the major problems.
Furthermore,

22
3_dataset.tex
View File

@ -4,7 +4,7 @@
In this study,
we leverage our previous dataset \textit{DupPR}~\cite{dup2018},
which %describes
contain the duplicate relations among pull requests and the profiles and review comments of pull requests from 26 popular OSS projects hosted on GitHub.
contains the duplicate relations among pull requests and the profiles and review comments of pull requests from 26 popular OSS projects hosted on GitHub.
We also extend the dataset by adding complementary data, including code commits, check statuses of DevOps tools and contribution histories of developers. %%%do you put the dataset somewhere?
% \subsection{Construction of \textit{DupPR}} %%%since this is in the existing work, there is no need to describe its construction process in such detail, you may describe what a record is in DupPR instead of the construction process.
@ -21,19 +21,19 @@ Item \texttt{idn\_cmt} is a review comment of either \texttt{pr1} or \texttt{pr2
which is used by reviewers to state the duplicate relation between \texttt{pr1} and \texttt{pr2}.
{\color{hltext}
The dataset meant to only contain the accidental duplicates
of which all the authors were not aware of other pull requests when creating their own pull requests.
of which all the authors were not aware of the other pull request when creating their own pull request.
In order to increase the accuracy of the dataset,
we recheck it again and filter out the intentional duplicates
that were not found before.
Specifically,
we omit duplicates from the dataset when they fit one of the following criteria:
\textit{i)} The authors' discussion on the associated issue reveals that the duplication is on purpose.
\textit{i)} The authors' discussion on the associated issue reveals that the duplication was on purpose.
A representative comment indicating intentional duplication is \textit{``I saw your PR and there wasn't any activity or follow up in that from last 18 days''};
\textit{ii)} One author has performed some kinds of actions on the other pull request before submitting her/his own.
\textit{ii)} One author performed some kinds of actions on the other pull request before submitting her/his own.
In addition to commenting,
actions like assigning/unassigning reviewers and adding/removing labels are also taken into consideration;
and \textit{iii)} One author immediately mention the other pull request after submitting her/his own,
which means that the author might already knew that pull request before.
actions like assigning reviewers and adding labels are also taken into consideration;
and \textit{iii)} One author immediately mentioned the other pull request after submitting her/his own,
which means that the author might already know that pull request before.
}
@ -103,14 +103,17 @@ while \textit{failure} indicates that the check has failed.
If the check is still running and no result is returned,
its \texttt{state} is \textit{pending}.
{\color{hltext}
\textit{error} indicates a check did not successfully run and produced a error.}
\textit{error} indicates a check did not successfully run and produced a error.
Following the guidelines of prior work~\cite{bel16oop,souza2017sentiment},
we treat the state error as the same as failure, which are both opposed to success.
}
The \texttt{context} indicates which tool is used in a specific check.
{\color{hltext}
According to the bot taxonomy defined in prior study~\cite{Wessel2018bot},
the checking tools can be classified into three categories:
CI (report continuous integration test results, \eg \texttt{continuous-integration/travis-ci}),
CLA (ensure license agreement signing \eg \texttt{cla/google}),
and RC (review source code \eg \texttt{coverage/coveralls} and \texttt{codeclimate}).
and CR (review source code \eg \texttt{coverage/coveralls} and \texttt{codeclimate}).
Based on all context-level check statuses of a commit,
the API also returns a overall check status of that commit~\cite{statusrule},
which we call the \textit{commit-level check status}.
@ -166,6 +169,7 @@ and who (\texttt{ISC.user\_id}) commented on that issue at what time (\texttt{IS
Based on this information
we can acquire the whole contribution history for a specific developer.
it mean that.
{\color{hltext}
\subsubsection{Poularity and reputation}

6
4_rq1.tex
View File

@ -263,9 +263,9 @@ Duplicate pull requests result in considerable redundancy in %code programming
writing code and evaluation.
On average, each group of %% each pair?
duplicate pull requests would result in
code patch redundancy of more than 17 files and 1,004 lines of code,
code-review redundancy of more than 7 review comments created by more than 2 reviewers,
and DevOps redundancy of more than 1 commit-level check and more than 3 context-level checks.
code patch redundancy of more than 18 files and 1,100 lines of code,
code-review redundancy of more than 5 review comments created by more than 2 reviewers,
and DevOps redundancy of more than 1 commit-level check and more than 2 context-level checks.
Moreover,
duplicate pull requests significantly slow down the review process, requiring more reviewers for extended discussions.
}

260
4_rq2.tex
View File

@ -14,14 +14,14 @@ For each relationship,
we investigate whether contributors' work patterns and their collaborating environment
have any flaw that may produce duplicate pull requests.
{\color{hltext}
% {\color{hltext}
Second,
we investigate the differences between duplicate and non-duplicate pull requests, as described in Section~\ref{ss:differenceanalysis},
in order to extract useful information that can provide insights into duplicates detection.
we investigate the differences between duplicate and non-duplicate pull requests, as described in Section~\ref{ss:differenceanalysis}.
% in order to extract useful information that can provide insights into duplicates detection.
We identify a set of metrics from prior studies to characterize pull requests.
We then conduct comparative exploration and regression analysis to
\hl{examine the characteristics that can distinguish duplicate with non-duplicate pull requests}.
}
examine the characteristics that can distinguish duplicate with non-duplicate pull requests.
% }
@ -198,12 +198,14 @@ some developers did not check that someone
has already submitted duplicate pull requests.
{\color{hltext}
For example,
many contributors said that they did not search the pull request list for similar work (\eg \textit{``Oh, Sorry I did not search for a previous PR before submitting a PR''} and \textit{``Ah should have searched first, thanks''})
or their search was not complete because they only searched the open pull requests and forgot the closed ones
quite a few contributors said that they did not search the pull request list for similar work (\eg \textit{``Oh, Sorry I did not search for a previous PR before submitting a PR''} and \textit{``Ah should have searched first, thanks''}).
In some cases,
developers' search was not complete because they only searched the open pull requests and missed the closed ones
(\eg \textit{``Ah, my bad. I thought I searched, but I must have only been looking at open''}).
}
The survey conducted by Gousios~\cite{gousios2016work} also showed that
45\% contributors occasionally or never check whether similar pull requests already exist before coding.
}
@ -287,10 +289,11 @@ after $mst\_pr$ has been submitted.
During that long period,
many new pull requests have been submitted in popular projects.
For example,
242 pull requests were submitted
during the 42 days between
\textit{symfony/symfony/\#18788} and \textit{symfony/symfony/\#19186}.
These pull requests can occupy 10 pages in the issue tracker,
307 pull requests were submitted
% during the 42 days
between
\textit{pandas-dev/pandas/\#9350} and \textit{pandas-dev/pandas/\#10074}.
These pull requests can occupy more than 10 pages in the issue tracker,
which makes it rather hard and ineffective to review historical pull requests page by page, as a developer stated
\textit{``...This is a dup of that PR. I should have looked harder as I didn't see that one when I created this one...''}.
@ -318,23 +321,24 @@ which makes it rather hard and ineffective to review historical pull requests pa
\noindent \textbf{Overlooking linked pull requests.}
When developers submit a pull request to solve an exiting GitHub issue,
they can build a link between the pull request and the issue by referencing the issue in the pull request description.
At the meantime,
the cross-reference is displayed in the discussion timeline of the issue.
The cross-reference is also displayed
in the discussion timeline of the issue.
Links not only allow pull request reviewers to find out the issue to be solved by a pull request
but also help developers who are concerned with an issue to discover which pull requests have been submitted for that issue.
In some cases,
contributors forgot to or did not examine the linked pull requests to an issue
to make sure that there is no ongoing work on that issue
contributors did not examine or did not notice the linked pull requests to an issue
(\eg \textit{``Uhm yeah, didn't spot the reference in \#21967''}
and \textit{``Argh, didn't see it in the original issue. Need more coffee I guess''}).
and \textit{``Argh, didn't see it in the original issue. Need more coffee I guess''})
to make sure that there is no ongoing work on that issue,
and consequently submit a duplicate pull request.
\vspace{0.5em}
\noindent \textbf{Lack of links.}
If a developer does not link her/his pull request to the associated issue,
other developers might consider that no patch has been pushed to fix that issue.
And consequently,
someone else might submit a duplicate pull request.
As a result,
another interested developer might submit a duplicate pull request.
For example,
a developer \textit{Dev2} submitted a pull request
\textit{facebook/react/\#6135} to address
@ -344,7 +348,7 @@ However,
a duplicate pull request \textit{facebook/react/\#6121} was already submitted by \textit{Dev1} before him.
The conversation between \textit{Dev1} and \textit{Dev2}
(\textit{Dev2} said \textit{``I'm glad to hear that. But please link your future PRs to the issues''}, and \textit{Dev1} replied \textit{``Yeah I will that's on me!''})
revealed that missing the issue link can account for the duplication.
revealed that the lack of the link can account for the duplication.
@ -355,12 +359,13 @@ they receive notifications about events that occur in the project,
\eg new commits and pull requests.
The notifications are displayed in developers' GitHub dashboard and,
if configured, sent to developers via email.
Sometimes,
developers might miss the notification about the submission of a similar pull request due to information overload~\cite{Dabbish2013Leveraging},
However,
developers might miss the notification about the submission of a similar pull request due to information overload~\cite{Dabbish2013Leveraging} or some other reasons,
and eventually submit a duplicate.
For example,
\textit{kubernetes/}
was duplicate of \textit{kubekubernetes/\#43902rnetes/kubernetes/\#43871}
\textit{kubernetes/kubernetes/\#43902}
was duplicate of
\textit{kubernetes/kubernetes/\#43871}
because the author of \textit{kubernetes/kubernetes/\#43902} \textit{``missed the mail for the PR it seems :-/''}.
}
@ -373,15 +378,15 @@ because the author of \textit{kubernetes/kubernetes/\#43902} \textit{``missed th
\vspace{0.5em}
\noindent \textbf{Unawareness of parallel work.}
{\color{hltext}
Many developers who encounter a problem prefer to fix the problem by
Developers who encounter a problem might prefer to fix the problem by
themselves and submit a pull request,
instead of reporting the problem in the issue tracker and waiting for a fix.
When a problem is encountered by two developers at the same time,
regardless of which developer is the first to start to work,
the other developer might also start to work before the first developer submits a pull request.
regardless of which developer is the first to work on the problem,
the other developer might also start to work on the problem before the first developer submits a pull request.
In such cases,
both developers are unaware of the concurrent activities of each other on the same problem,
because their local work is conducted locally and is not publicly visible.
both developers are unaware of concurrent activities of each other,
because their local work is conducted offline and is not publicly visible.
For example,
the authors of \textit{emberjs/ember.js/\#4214} and \textit{emberjs/ember.js/\#4223}
individually fixed the same typos in parallel without being aware of each other,
@ -394,16 +399,16 @@ and finally submitted two duplicate pull requests.
{\color{hltext}
\vspace{0.5em}
\noindent \textbf{Implementing without claiming first.}
In some cases,
a developer might directly start to implement a patch for a GitHub issue
without claiming that issue first (\ie leaving a comment like \textit{``I'm on it''}).
Sometimes,
developer directly start to implement a patch for a GitHub issue
without claiming the issue first (\ie leaving a comment like \textit{``I'm on it''}).
This can introduce a risk that
other interested developers might also start to work on the same issue
without awareness of that there is actually already a developer working on that issue.
For example,
although two developers were both trying to solve the issue
\textit{facebook/react/\#3948},
neither of them claimed the issue before coding their patches.
neither of them claimed the issue before coding their patch.
Finally, they submitted two duplicate pull requests
\textit{facebook/react/\#3949} and \textit{facebook/react/\#3950}.
The phenomenon that developers are not used to claim issues was also reported in previous research~\cite{zhou2019fork}.
@ -414,10 +419,10 @@ The phenomenon that developers are not used to claim issues was also reported in
\noindent \textbf{Overlooking existing claims.}
For a GitHub issue
which has been already claimed by a developer,
if other developers forget to check or do not notice the existing claim comment in the issue discussions,
they would prepare duplicate patches.
if other developers do not check or do not notice the existing claim comment in the issue discussions,
they can prepare duplicate patches.
For example,
a developer \textit{Dev1} first claimed \textit{scikit-learn/scikit-learn/\#8503} by
a developer \textit{Dev1} first claimed the isssue \textit{scikit-learn/scikit-learn/\#8503} by
leaving a comment \textit{``We are working on this''}.
However,
another developer \textit{Dev2} did not notice this claim
@ -431,7 +436,8 @@ and submitted two duplicate pull requests
\vspace{0.5em}
\noindent \textbf{Communication failure.}
Sometimes,
% miscommunication?
In some cases,
developers communicate in person or online
about a software problem or a desired feature.
However,
@ -439,7 +445,7 @@ their communication might fail,
which does not reach a clear agreement on
who takes the responsibility of submitting a patch.
As a result,
more than one developer might submit a patch, resulting in duplicates.
more than one developer might submit a patch, leading to duplicates.
For example,
two duplicate pull requests
\textit{saltstack/salt/pull/\#17691}
@ -463,7 +469,7 @@ and (b) $INC$, which includes $dup\_pr$ of each pair of inclusive duplicates.
Figure~\ref{fig:dur_cmp} plots the duration statistics of each group together along the group \textit{NON}, which includes all non-duplicate pull requests.
We observe that
compared with the pull requests in \textit{NON},
the pull requests in $OVL$ have longer local durations
the pull requests in $OVL$ and $INC$ have longer local durations
regarding median measures.
We also find that the difference between \textit{NON} and $OVL$ is significant
according to {\color{hltext}the $\widetilde{\textbf{T}}$-procedure test results,
@ -510,15 +516,15 @@ As discussed in \cite{gousios2016work}, developers rarely recheck the existence
\noindent
\textit{
\textbf{RQ2-1:}
\hl{We identified 11 contexts where duplicate pull requests occur,
We identified 11 contexts where duplicate pull requests occur,
which are mainly relating to
developers' behaviors,
e.g., not checking for existing work, not claiming before coding,
not providing traceability links, and overlong local work,
not providing links, and overlong local work,
and their collaborating environment,
e.g., unawareness of parallel work, lack of effective tools for checking for duplicates,
Communication failure and missing notifications
can also result in duplicates.}
e.g., unawareness of parallel work, missing notifications, lack of effective tools for checking for duplicates.
Communication failure
can also result in duplicates.
% missing notifications,
% not checking for existing work,
% large searching space,
@ -562,11 +568,12 @@ we want to observe what distinguishing characteristics of duplicate pull request
% effect of the features on pull requests' likelihood of being duplicates.
{\color{hltext}
First,
we conduct a literature review and identify \hl{25} metrics
that are used in prior research relating \hl{to OSS development},
we
% conduct a literature review and
identify metrics that are used in prior research,
as shown in Section~\ref{ss:metrics}.
Then,
we compare duplicate and non-duplicate pull requests in terms of each metric
we compare duplicate and non-duplicate pull requests in terms of each metric,
and check whether significant difference could be observed between them
through statistical test,
as described in Section~\ref{ss:compare_explore}.
@ -579,15 +586,14 @@ between the collected metrics and pull requests' likelihood of being duplicates.
\subsubsection{Metrics}
\label{ss:metrics}
% 说一下一些指标是截止到PR submission
We conduct a literature review to identify the metrics that have been studied in prior studies.
We conduct a literature review to identify the metrics that have been studied in prior research.
We select influential research papers in the areas of patch submission and acceptance~\cite{jiang2013will,hassan2009predicting,Baysal2015Inv,Pham2013Creating,weissgerber2008small}
and pull request development~\cite{Tsay2014Influence,Gousios:2014,yu16det,zhou2019fork,Rahman2014An,vasilescu2016sky,Bor17Und,Kononenko2018Studying,vasilescu2015gender},
which have been recently published in the leading software engineering journals and conferences,
\eg TSE, ICSE and FSE.
After analyzing these papers,
we identify \hl{25} metrics that can be computed at pull request submission time.
we identify metrics that can be computed at pull request submission time.
The identified metrics are classified into the following three categories:
@ -604,18 +610,18 @@ as an indicator of the project maturity~\cite{Tsay2014Influence,yu16det,Rahman20
However,
a project does not necessarily use the pull request model in the first place.
We also use the metric \texttt{prmodel\_age} to indicate
how long a project has adopted the pull request development model.
how long a project has used the pull request development model.
\vspace{0.2em}
\textit{Workload.}
\textit{\hl{Workload.}}
The discussion of issues and pull requests might cost days to months to come to an end.
At any given time,
a bunch of open issues and pull requests
might be discussed simultaneously.
Prior studies have characterized project integrators' workload using two metrics:
Prior studies have characterized project \hl{integrators' workload} using two metrics:
\texttt{open\_tasks}~\cite{yu16det} and \texttt{team\_size}~\cite{Tsay2014Influence,Gousios:2014,yu16det},
which are the number of open issues and open pull requests at the pull request submission and
which are the number of open issues and open pull requests at the pull request submission time and
the number of active core team members during the last three months, respectively.
@ -623,11 +629,11 @@ the number of active core team members during the last three months, respectivel
\textit{Popularity.}
In measuring project popularity,
the metric \texttt{stars},
\ie the number of stars the project has got,
\ie the total number of stars the project has got prior to the pull request submission,
was commonly used in prior studies~\cite{Bor17Und,Tsay2014Influence}.
In addition,
we also considered three other popularity-related metrics:
\texttt{forks}, \texttt{prs}, and \texttt{contributors},
\texttt{forks}, \texttt{pullreqs}, and \texttt{contributors},
which are the number of forks, the number of pull requests, and the number of contributors
of the project, respectively.
@ -654,14 +660,14 @@ Developers' experience before they submit the pull request has been analyzed in
This measure can be computed from two perspectives:
project-level experience and community-level experience.
The former measures the number of previous pull requests
that have submitted to a specific project (\texttt{prev\_pullreqs\_proj}) and their acceptance rate (\texttt{prev\_prs\_acc\_proj}).
that the developer have submitted to a specific project (\texttt{prev\_pullreqs\_proj}) and their acceptance rate (\texttt{prev\_prs\_acc\_proj}).
The latter measures the number of previous pull requests
that have been submitted to GitHub (\texttt{prev\_pullreqs}) and their acceptance rate (\texttt{prev\_prs\_acc}).
that the developer have submitted to GitHub (\texttt{prev\_pullreqs}) and their acceptance rate (\texttt{prev\_prs\_acc}).
When calculating acceptance rate,
the determination of whether the pull request was integrated
through other mechanisms than GitHub's merge button follows the heuristics defined in previous studies~\cite{Gousios:2014,zhou2019fork}.
We also use two metrics \texttt{first\_pr\_proj} and
\texttt{first\_pr} to represent whether the pull request is the first one submitted by a developer to a specific project and GitHub, respectively.
\texttt{first\_pr} to represent whether the pull request is the first one submitted by the developer to a specific project and GitHub, respectively.
\vspace{0.2em}
\textit{Standing.}
@ -671,7 +677,7 @@ was commonly used as a signal of the developer's standing within the project~\ci
Furthermore,
a continuous metric \texttt{followers},
\ie the number of GitHub users that are following the pull request submitter,
was used to represent the developers' standing within the community~\cite{Tsay2014Influence,Gousios:2014,yu16det}.
was used to represent the developers' standing in the community~\cite{Tsay2014Influence,Gousios:2014,yu16det}.
\vspace{0.2em}
@ -697,7 +703,7 @@ Prior studies~\cite{Tsay2014Influence,Gousios:2014,vasilescu2016sky} quantified
in different granularity.
The commonly used metrics are
% the number of commits (\texttt{commits}),
the number of changed files (\texttt{files}) and the number of changed lines of codes added and deleted (\texttt{churn}).
the number of changed files (\texttt{files}) and the number of changed lines of codes added and deleted (\texttt{loc}).
\vspace{0.2em}
\textit{Textual length.}
@ -724,7 +730,8 @@ Prior studies~\cite{mockus2000identifying,hassan2009predicting} summarized that
\textit{feature introduction (FI)},
and \textit{general maintenance (GM)}.
The change type (\texttt{change\_type}) of the pull request
is identified based on a set of manually verified keywords~\cite{mockus2000identifying}.
is identified by analyzing its description
based on a set of manually verified keywords~\cite{mockus2000identifying}.
Prior studies~\cite{hindle2007release,vasilescu2014variation} also identified the types of developers' activities
on the basis of the types of changed files.
We follow the classification by Hindle \etal~\cite{hindle2007release},
@ -733,7 +740,7 @@ which include four types:
\textit{Test} changing test files,
\textit{Build} changing build files,
and \textit{Doc} changing documentation files.
This metric (\texttt{activity\_type}) is determined by checking the names and extensions of changed files.
This metric (\texttt{activity\_type}) is determined by checking the names and extensions of the files changed by the pull request.
@ -751,8 +758,7 @@ we use the following hypotheses:
% $H_{0}$: duplicate pull requests exhibit a value of metric $m$ equal to that one exhibited by non-duplicate pull requests.
\vspace{0.2em}
$H_{0}$: duplicate
and non-duplicates pull requests exhibit the same value of metric $m$.
$H_{0}$: duplicate and non-duplicates pull requests exhibit the same value of metric $m$.
\vspace{0.2em}
$H_{1}$: duplicate
@ -779,7 +785,7 @@ $\forall m \in$
\vspace{0.2em}
$H_{0}$ is tested with \textit{Mann-Whitney-Wilcoxon} test~\cite{Wilcoxon1945Individual} on continuous metrics and \textit{Chi-square} test~\cite{ramsey2012statistical} on categorical metrics.
The test results are listed in Table~\ref{tab:hypo_test}
which reports the and p-value and effect size of each test.
which reports the p-value and effect size of each test.
To measure the effect size,
we use \textit{Cliff's delta (d)}~\cite{Long2003Ordinal} as it is a non-parametric approach
which does not require the normality assumption of a distribution.
@ -799,13 +805,15 @@ This means that
duplicate and non-duplicate pull requests
are significantly different in terms of all metrics except for \texttt{open\_tasks}.
Following the previous guidelines~\cite{Thongtanunam2017Revisiting,Ponzanelli2017Supporting} on interpreting the effect size
(trivial: $0.147\le |d|$; small: $0.147<|d|<0.33$; medium: $0.33 \le |d|<0.474$; large: $|d| \ge 0.474$),
\hl{we find that the effect size of difference is generally small with a maximum of 0.285.
Specifically,
for most submitter-level metrics,
the observed effect size is small,
and for most project-level and patch-level metrics,
the observed effect size is negligible.}
(trivial: $|d|\le 0.147$; small: $0.147<|d|<0.33$; medium: $0.33 \le |d|<0.474$; large: $|d| \ge 0.474$),
% \hl{
we find that the effect size of difference is generally small with a maximum of 0.285.
% Specifically,
% for most submitter-level metrics,
% the observed effect size is small,
% and for most project-level and patch-level metrics,
% the observed effect size is negligible.
% }
@ -900,7 +908,7 @@ we apply a regression analysis to model the effect of the selected metrics on pu
\vspace{0.5em}
\noindent\textbf{Regression modeling.}
We build a \hl{mixed effect logistic regression model}
We build a mixed effect logistic regression model
which is fit to capture the relationship between the explanatory variables, \textit{i.e.}, the metrics discussed in Section~\ref{ss:metrics}, and a response variable, \textit{i.e.}, \textit{is\_dup}, which indicates whether a pull request is a duplicate.
Instead of building one model with all metrics at once,
we add one level metrics at a time and build a model,
@ -910,12 +918,9 @@ we compare the fit of three models:
a)~\textit{Model 1}, which includes only project-level variables,
b)~\textit{Model 2}, which adds the submitter-level variables,
and c)~\textit{Model 3}, which adds patch-level variables.
To avoid project-level effects unrelated to the explanatory variables,
To avoid \hl{project level effects} unrelated to the explanatory variables,
the selected metrics are modeled as fixed effects, and
a new variable \textit{proj\_id} is modeled as a random effect.
a new variable \texttt{proj\_id} is modeled as a random effect.
In the model,
all numeric factors are log transformed (plus 0.5 if necessary)
to stabilize variance and reduce heteroscedasticity~\cite{metz1978basic}.
@ -926,7 +931,7 @@ regression models, but ensures that our models are robust against outliers~\cite
%which probably decreases the models' robustness because of high leverage.
In addition,
we check for the Spearman correlation of coefficients and the Variance Inflation Factors (VIF below 5 as recommended~\cite{cohen2014applied}) among predictors to overcome the effect of multicollinearity.
This process leaves us with \hl{17} features,
This process leaves us with 17 features,
which can be
seen in Table~\ref{tab:diff_models}.
% \hl{TODO: necesseary to add a metric overview table?}
@ -1002,68 +1007,71 @@ we discuss their effects based on Model 3.
As for project level predictors,
With regard to project-level predictors,
\texttt{open\_tasks} and \texttt{team\_size} have significant, positive effects.
This suggests that
the more open tasks (pull requests and issues) and active core team members at the pull request submission time,
the more likely a pull request is a duplicate.
We argue that \hl{****}.
We assume that the more open tasks the more hard for developers to check for similar work,
and the more active core member the more possible external contributors would collide with them.
Perhaps surprisingly,
the predictor \texttt{watchers} has a strong, negative effect,
which means that
the more popular a project is the less likely a pull request is a duplicate.
One possible explanation is that
which means that the more popular the project becomes
the less likely the submitted pull request is a duplicate.
\hl{One possible explanation is that
when the project popularity increases,
both the number of duplicate and the number of non-duplicate pull requests tend to increase,
however, the ratio of increased non-duplicate pull requests is higher than that of duplicate pull requests.
.
both the numbers of duplicate and non-duplicate pull requests tend to increase,
however, the ratio of increased non-duplicate pull requests is higher than that of duplicate pull requests.}
We also see a negative effect of the predictor \texttt{hotness}.
This indicates that
changing cold files increases the possibility of pull requests being duplicates.
we assume that pull requests changing hots files tend to be reviewed faster and get accepted in a timely fashion.
\hl{We assume that pull requests changing hots files tend to be reviewed faster and get accepted in a timely fashion.}
A quick review can allow the target issue to be solved in short time,
which prevents others from encountering the same issue and submitting duplicate pull requests.
With regard to submitter level predictors,
two predictors relating to developer experience present opposite effects,
that is,
the predictor \texttt{prev\_prs\_acc} has a negative effect
while \texttt{first\_pr\_proj TRUE} has a positive effect.
This exactly suggests that
pull requests submitted by inexperienced developers
As for submitter-level predictors,
\texttt{prev\_prs\_acc} has a negative effect
and \texttt{first\_pr\_proj} has a positive effect when its value is \texttt{TRUE}.
This suggests that
pull requests submitted by inexperienced developers and newcomers
are more likely to be duplicates.
\hl{****more talk?****}
One possible explanation may be that
inexperienced new developers lack sufficient knowledge in the collaborative development process and practices~\cite{steinmacher2015social},
and cannot effectively avoid duplicated work.
We notice that
the predictors \texttt{core\_team True} and \texttt{prior\_interaction} have significant, negative effects,
indicating that
the predictors \texttt{prior\_interaction}
and \texttt{core\_team} (\texttt{TRUE}) have significant, negative effects.
This indicates that
pull requests from core team members and developers
who are with a stronger social connection to the project
who have a stronger social connection to the project
are less likely to be duplicates.
\hl{****more talk?****}
This might be due to their well-maintained awareness of the project status
coming from their active participation in the project~\cite{Gutwin2004Group}.
For patch level predictors,
two size related predictors present different effects.
The predictor \texttt{loc} has negative effects,
For patch-level predictors,
two size related predictors present opposite effects.
The predictor \texttt{loc} has a negative effect,
which indicates that
pull requests changing more lines of codes have a less chance of being duplicates.
While the predictor \texttt{files\_changed} has a positive effect,
suggesting that
pull requests changing more files are more likely to be duplicates.
We argue that \hl{*****}.
The predictor \texttt{text\_len} has a positive effect.
This indicates pull requests with complex description
Currently, \hl{it is unclear about the cause of opposite effects}.
We think this interesting result deserves future investigation.
The predictor \texttt{text\_len} has a positive effect,
indicating that pull requests with complex description
are more likely to be duplicates.
\hl{****more talk?****}
The predictor \texttt{issue\_tag TRUE} also has a positive effect,
suggesting that pul requests associated with existing issues or pull requests have a greater chance of being duplicates.
Longer description may indicate higher complexity and thus longer evaluation~\cite{yu16det}
which increases the likelihood of the same issue being encountered by more developers who might also submit a patch for the issue.
The predictor \texttt{issue\_tag} has a positive effect when its value si \texttt{TRUE},
suggesting that pull requests solving already tracked issues have greater chances of being duplicates.
One possible reason is that
tracked issues are already publicly visible, and
they are more likely to attract more developers and thus result in conflicts.
In terms of change types,
they are more likely to attract more interested developers and result in conflicts.
In terms of change types (\texttt{change\_type}),
we can see that
compared with pull requests of the type \textit{FR},
pull requests of the type \textit{FI}, \textit{GM} and \textit{Other} are less likely to be duplicates.
@ -1071,13 +1079,13 @@ We speculate that fixing bugs are more likely to produce duplicates
because bugs tend to have general effect on a bigger developer base
compared to new feature or maintenance requirements
which might be specific to a certain group of developers.
For activity types,
For activity types (\texttt{activity\_type}),
we notice that
pull requests changing test files (\texttt{activity Test}) and documentation files (\texttt{activity Doc})
have a less chance of being duplicates,
have less chances of being duplicates,
compared with those changing source code files.
OSS projects usually encourage newcomers to try their first contribution by writing documentation and test cases~\cite{TD-guide, TD-label}.
We conjecture that activities changing sourced code files might require more effort and time to conduct the local work,
We conjecture that activities changing source code files might require more effort and time to conduct the local work,
which are more risky and prone to duplication.
}%for red color
@ -1091,13 +1099,17 @@ which are more risky and prone to duplication.
\textbf{RQ2-2:}
Duplicate pull requests are significantly different
from non-duplicate pull requests in terms of
whether project-level, submitter-level, or patch-level characteristics.
% The popularity, workload, and area hotness of the project;
% the experience, standing status, and social connection to the project of the submitter;
% and the size, issue visibility, and type of the patch
% can significantly influence the likelihood of pull requests being duplicates.
Pull requests submitted when the project has high popularity \hl{and} low workload, \hl{and} modifying recently modified code are less likely to be duplicates.
Pull requests from core team members and developers with more contribution experience \hl{and} strong social connection to the project are less likely to be duplicates.
Pull requests changing more files with fewer commits and codes, changing source code files, fixing bugs, of complex description, \hl{and} associated with existing tasks are more likely to be duplicates.
project-level characteristics (
e.g., changing cold files and submitted when the project has more active core team members),
submitter-level characteristics (
e.g., submitted from developers who have less contribution experience and developers who have weaker connection to the project),
and patch-level characteristics (
e.g., solving already tracked issues rather than non-tracked issues and fixing bugs rather than adding new features or refactoring).
% Duplicate pull requests are significantly different
% from non-duplicate pull requests in terms of
% whether project-level, submitter-level, or patch-level characteristics.
% Pull requests submitted when the project has high popularity \hl{and} low workload, \hl{and} modifying recently modified code are less likely to be duplicates.
% Pull requests from core team members and developers with more contribution experience \hl{and} strong social connection to the project are less likely to be duplicates.
% Pull requests changing more files with fewer commits and codes, changing source code files, fixing bugs, of complex description, \hl{and} associated with existing tasks are more likely to be duplicates.
}
\end{mdframed}

373
4_rq3.tex
View File

@ -6,14 +6,12 @@ we first construct a dataset of the integrators' choice between duplicates, as d
Then we perform two investigations.
First,
given that prior wok~\cite{yu16det,Gousios:2014} has found several factors, \textit{e.g.,} the size and comments of pull requests and the role and experience of contributors,
have effects on the acceptance of pull requests,
we examine their effects on the competition among duplicate pull requests.
As described in Section~\ref{ss:reg-ana},
we consider a total of 12 variables from three dimensions
as described in Section~\ref{ss:reg-ana},
we identify metrics from prior work in the area of patch evaluation and acceptance,
and apply them in a regression model to analyze their effects on the response variable \textit{accept},
which indicates whether a duplicate pull request has been accepted or rejected.
Second,
we want to learn what exactly integrators examine
when they accept a duplicate pull request rather than its counterparts.
@ -48,11 +46,10 @@ as elaborated in the following sections.
In the study,
we focus on the duplicate pairs
that integrators compared after detecting their duplicate relation.
{\color{hltext}
First,
we exclude duplicate pairs in which both pull requests remain open.
Then, we exclude duplicate pairs which were closed but not compared by integrators.
Figure~\ref{fig:filter} shows different cases where the relation identification and decision-making between duplicates happened at different times.
Figure~\ref{fig:filter} shows the different cases where the relation identification and decision-making between duplicates happened at different times.
In case A, $mst\_pr$ was closed before $dup\_pr$ was submitted.
In case B,
although $mst\_pr$ was closed after $dup\_pr$ was submitted,
@ -64,12 +61,11 @@ because integrators did not make a comparison between duplicates before making d
Moreover,
we also exclude from duplicate pairs of case C those in which one pull request was closed by its submitter before integrators leave any comment.
Finally,
we exclude \hl{738} non-compared duplicate pairs, and \hl{1,197} duplicate pairs remain.
}
we exclude {\color{hltext}738} non-compared duplicate pairs, and {\color{hltext}1,197} duplicate pairs remain.
\begin{figure}[h]
\centering
\includegraphics[width=0.49\textwidth]{resources/filter.pdf}
\includegraphics[width=0.45\textwidth]{resources/filter.pdf}
\caption{Different cases where the relation identification and decision-making between duplicates happen at different times.}
\label{fig:filter}
\end{figure}
@ -152,17 +148,16 @@ we {\color{hltext}454} duplicate pairs in which only one pull request is accepte
Additionally,
it is possible that both duplicates in a pair are rejected.
For example,
\hl{
\textit{rails/rails/\#10737} and \textit{rails/rails/\#10738}} were both rejected by integrators
\textit{rails/rails/\#10737} and \textit{rails/rails/\#10738} were both rejected by integrators
because integrators think the change is not necessary.
In total,
we find {\color{hltext}29} duplicate pairs in which both pull requests are rejected.
Finally,
as shown in Table~\ref{tab:rq3-dst-bnm},
we collect a total of {\color{hltext}1,175} duplicate pairs in which only one is accepted.
we collect a total of {\color{hltext}1,132} duplicate pairs in which only one is accepted.
Subsequently,
we conduct regression analysis and manual inspection based on those {\color{hltext}1,175} duplicate pairs.
we conduct regression analysis and manual inspection based on those {\color{hltext}1,132} duplicate pairs.
\begin{table}[h]
@ -193,8 +188,8 @@ we present the selected predictors, regression models and analysis results.
{\color{hltext}
The predictor selection is based on prior work in the area of patch acceptance analysis~\cite{Gousios:2014,yu16det,gousios2015work,Dabbish2012Social,weissgerber2008small,Tsay2014Influence,Baysal2015Inv,Kon15Inv,Kononenko2018Studying,Bosu2016Process,Pham2013Creating,jiang2013will}.
The selected predictors are split into three categories:
patch-level metrics,
submitter-level metrics,
patch-level metrics,
and review-level metrics.
Compared with the study in Section~\ref{ss:differenceanalysis},
this study additionally includes review-level metrics because
@ -202,7 +197,7 @@ many signals in the code review process, \eg review discussion length,
are available after pull request submission, and
past studies have found that review-level metrics have significant effects on pull request acceptance~\cite{yu16det,Gousios:2014,jiang2013will,Kononenko2018Studying}.
However,
this study does not include project-level metrics
this analysis does not include project-level metrics
because two duplicate pull requests have the same project environment at the decision-making time.
@ -210,29 +205,27 @@ because two duplicate pull requests have the same project environment at the dec
\vspace{0.5em}
\noindent\textbf{Submitter-level metrics.}
% experience
% 这个justification可能有点不太合适，应该是experience高了，更有可能提交高质量的PR
Integrator might examine contributors' track record within the project or other projects when assessing the quality of pull requests~\cite{gousios2015work}.
Developer experience is an important factor affecting patch acceptance.
Prior studies~\cite{Baysal2015Inv,Gousios:2014,Kononenko2018Studying} have shown that
having more contribution experience increases the likelihood of pull request acceptance.
As discussed in Section~\ref{ss:differenceanalysis},
we still use six metrics,
more experienced developers are more likely to get their patches accepted.
To understand whether developer experience affects integrators' choice between duplicates,
we still use six metrics to operationalize developer experience,
\ie \texttt{prev\_prs\_porj}, \texttt{prev\_acc\_proj}, \texttt{prev\_prs}, \texttt{prev\_acc}, \texttt{first\_pr\_proj} and
\texttt{first\_pr},
to operationalize contribution experience for developers.
as discussed in Section~\ref{ss:differenceanalysis}.
% standing
Prior studies also showed that pull requests submitted by developers with high standing in the project or the general community are more likely to be accepted~\cite{Tsay2014Influence,yu16det}.
To investigate whether developers' standing affects integrators' choice between duplicates,
we include two metrics \texttt{core\_team} and \texttt{followers} defined in Section~\ref{ss:differenceanalysis}.
The previous studies showed that pull requests submitted by developers with high standing in the project and the general community are more likely to be accepted~\cite{Tsay2014Influence,yu16det}.
To investigate the influence of developers' standing on integrators' choice,
we include two metrics \texttt{core\_team} and \texttt{followers} defined in Section~\ref{ss:differenceanalysis}.
% social
A developer's social relationship and interaction history with others
can affect others' judgement for the developer's work~\cite{gousios2015work,Bosu2016Process,Pham2013Creating}.
Prior studies have found that
pull requests from developers with a stronger social connection to the project and integrators have a higher chance to be accepted~\cite{Tsay2014Influence,yu16det}.
Therefore,
in addition to the metric \texttt{prev\_interaction} as defined earlier,
we include the metric \texttt{social\_strength} to
represent the proportion of decision-making group members that have co-occurred with the submitter in at least one discussion during the last three months.
pull requests from developers with a stronger social connection to the project have a higher chance to be accepted~\cite{Tsay2014Influence,yu16det}.
In addition to the metric \texttt{prev\_interaction} as defined earlier,
we include the metric \texttt{social\_strength},
which represents the proportion of decision-making group members that have co-occurred with the submitter in at least one discussion during the last three months,
to examine the effects of social metrics on integrators' choice.
@ -249,37 +242,38 @@ represent the proportion of decision-making group members that have co-occurred
% \textit{Size.}
Prior studies~\cite{Gousios:2014,yu16det,Tsay2014Influence} have found that
large pull requests changing more files are less likely to be accepted.
large pull requests are less likely to be accepted.
Integrators value small pull requests as they are easy to assess and integrate~\cite{gousios2015work}.
To investigate the effect of patch size on integrators' choice between duplicates,
we include the three size-related metrics presented in Section~\ref{ss:differenceanalysis},
\ie \texttt{commits, \texttt{files}, \texttt{loc}}.
we include the two size-related metrics presented in Section~\ref{ss:differenceanalysis},
\ie \texttt{\texttt{files\_changed} and \texttt{loc}}.
% \textit{Length.}
% \textit{Issue\_tag}
The study conducted by Yu \etal~\cite{yu16det} showed that pull requests with complex description have a higher chance to be rejected.
It also revealed that pull requests containing links to issues have higher acceptance rate.
The study conducted by Yu \etal~\cite{yu16det} showed that pull requests with longer description have higher chances to be rejected.
It also revealed that pull requests containing links to issues have higher acceptance rates.
For this,
we include two description-related metrics, \ie \texttt{title\_len} and \texttt{desc\_len}, and \texttt{issue\_tag},
we include the metrics \texttt{text\_len} and \texttt{issue\_tag},
which are already defined in Section~\ref{ss:differenceanalysis}.
% \textit{test\_inclusion}
From the prior work~\cite{Dabbish2012Social,gousios2015work},
we learn that the existence of testing code
is treated as a positive signal when integrators evaluate pull requests.
Prior quantitative analysis~\cite{Gousios:2014,Tsay2014Influence, yu16det} also presented that
pull request with test cases are more likely to be accepted.
Therefore,
Pull requests with test cases are more likely to be accepted~\cite{Gousios:2014,Tsay2014Influence, yu16det}.
To investigate this,
we include a dichotomous metric \texttt{test\_inclusion} to indicate
whether the pull request has changed test files.
% \textit{hotness}
Gousios \etal~\cite{Gousios:2014} found that
whether the pull request modifies recently modified code
significantly influences its acceptance.
\hl{Even two pull requests are duplicate of each other,
they do not necessarily change the same files.}
Therefore, we include the metric \texttt{hotness} discussed in Section~\ref{ss:differenceanalysis}
to verify whether hotness would affect integrators' choice between duplicates.
Finally,
in the context of decision-making of duplicate pull requests,
% % \textit{hotness}
% Gousios \etal~\cite{Gousios:2014} found that
% whether the pull request modifies recently modified code
% significantly influences its acceptance.
% \hl{Even two pull requests are duplicate of each other,
% they do not necessarily change the same files.}
% Therefore, we include the metric \texttt{hotness} discussed in Section~\ref{ss:differenceanalysis}
% to verify whether hotness would affect integrators' choice between duplicates.
Finally,
in the context of
% decision-making of duplicate pull requests,
selection between duplicate pull requests,
we conjecture that the order that pull requests arrive
might affect integrators' choice.
The duplicate pull request submitted early might be more likely to be accepted than the late one,
@ -294,31 +288,41 @@ whether the pull request is submitted earlier than its counterpart.
\noindent\textbf{Review-level metrics.}
% human review
The participation of and feedback from human reviewers
has been shown to affect the review latency and outcome of pull requests~\cite{yu16det,Tsay2014Influence,Kononenko2018Studying}.
The participation metrics relating to human reviewers
have been shown to affect the latency and outcome of pull request review~\cite{yu16det,Tsay2014Influence,Kononenko2018Studying}.
The study by Tsay \etal~\cite{Tsay2014Influence}
showed that pull requests with higher amount of comments
showed that pull requests with a higher amount of comments
are less likely to be accepted.
A recent study~\cite{Kononenko2018Studying} also revealed that the number of participants leaving comments on the pull request affects its acceptance.
Accordingly,
To investigate the effects of human participation metrics on integrators' choice betw,
we include two comment-related metrics,
\ie \texttt{comments} and \texttt{comments\_inline} indicating the number of comments on pull request and the number of inline comments on source code, respectively,
\ie the total number of comments on the pull request (\texttt{comments})
and the number of inline comments on the source code (\texttt{comments\_inline}),
and two participant-related metrics,
\ie \texttt{comments\_devs} and \texttt{comments\_inline\_devs}
indicating the number of developers leaving the comments and
the number of developers leaving the inline comments, respectively.
Developers might leave affective comments to support or oppose a pull request according to whether they like the change~\cite{Tsa14Let,yu16det}.
To explore the relationship between discussion sentiment and integrators' choice,
we include metrics \texttt{comments\_neg} and \texttt{comments\_pos}
which are the proportions of comments expressing negative sentiment and positive sentiment, respectively.
\ie the number of developers leaving the comments (\texttt{comments\_devs}) and
the number of developers leaving the inline comments (\texttt{comments\_inline\_devs}).
% \hl{
Developers might leave affective comments
on a pull request
according to whether they like or dislike the pull request~\cite{Tsa14Let,Adams2014Do}.
% }
% \hl{
We speculate that duplicate pull requests receiving more positive comments than negative comments are more likely to be accepted.
To verify our hypothesis,
we include two metrics:
the proportion of the comments expressing positive sentiment (\texttt{comments\_pos})
and
the proportion of the comments expressing negative sentiment (\texttt{comments\_neg}).
% }
To analyze the sentiment in pull request comments,
we use the exiting state-of-the-art sentiment analysis tool Senti4SD~\cite{Calefato2017Sentiment} retrained on the latest GitHub data~\cite{Novielli2020Can}.
% auto tool review
Integrators also rely on automated testing tools to check pull request quality~\cite{gousios2015work}.
For example, prior study~\cite{yu16det} found that the presence of test failure has a negative effect on pull request acceptance.
We include three metrics \texttt{CI}, \texttt{CLA}, and \texttt{RC}, to represent the check statuses of the most recent commit in the pull request,
For example, prior study~\cite{yu16det} found that the presence of CI test failure has a negative effect on pull request acceptance.
To investigate this,
we include three metrics \texttt{CI}, \texttt{CLA}, and \texttt{RC}, to represent the check statuses of the most recent commit in the pull request,
which are returned by the three kinds of check tools discussed in Section~\ref{sec:dsrq}.
We also include a metric \texttt{final\_status} to represent the overall commit-level check status.
% We also include a metric \texttt{final\_status} to represent the overall commit-level check status.
% revision
Finally,
we include a metric \texttt{revisions},
@ -327,6 +331,7 @@ to indicate the maturity of the pull request.
The prior study~\cite{jiang2013will} showed that patch maturity is one of the major factors affecting patch acceptance.
All metrics are calculated at the time
integrators make the decision between duplicates.
The decision-making group involved in some metrics' calculation (\eg \texttt{social\_strength})
@ -349,9 +354,9 @@ to the time the decision is made.
\texttt{prev\_pullreqs}& 1.7e2 &3.1e2 &0&45.00&3.5e3\\
\texttt{prev\_prs\_acc}& 0.45 &0.26 &0.0&0.47&1.0\\
\texttt{first\_pr\_proj}& 0.29 &0.45 &0&0.00&1\\
\texttt{first\_pr}& 0.02 &0.15 &0&0.00&1\\
% \texttt{first\_pr}& 0.02 &0.15 &0&0.00&1\\
\texttt{core\_team}& 0.33 &0.47 &0&0.00&1\\
\texttt{prior\_interaction}& 1.0e3 &2.9e3 &0&43.00&4.3e4\\
% \texttt{prior\_interaction}& 1.0e3 &2.9e3 &0&43.00&4.3e4\\
\texttt{social\_strength}& 0.62 &0.46 &0.0&1.00&1.0\\
@ -370,7 +375,7 @@ to the time the decision is made.
\texttt{comments\_inline}& 1.55 &5.42 &0&0.00&1.1e2\\
\texttt{comments\_pos}& 0.17 &0.27 &0.0&0.00&1.0\\
\texttt{comments\_neg}& 0.04 &0.13 &0.0&0.00&1.0\\
\texttt{BT}& 0.10 &1.05 &-1.0&0.00&3.0\\
\texttt{CI}& 0.10 &1.05 &-1.0&0.00&3.0\\
\texttt{CLA}& -0.81 &0.60 &-1.0&-1.00&3.0\\
\texttt{CR}& -0.84 &0.56 &-1.0&-1.00&3.0\\
@ -384,31 +389,28 @@ to the time the decision is made.
}
\subsubsection{Statistical Analysis}
\label{ss:choice_reg}
Our goal is to explain the relationship (if any)
between the selected factors and the binary response variable
$accept$,
which indicates whether a duplicate pull request has been accepted over its counterpart (\texttt{1} for accepted and \texttt{0} for not accepted).
We use conditional logistic regression~\cite{connolly1988cd},
implemented in the \textit{survival} package of \textbf{R},
We use {\color{hltext}conditional logistic regression~\cite{connolly1988cd} with a random effect,
implemented in the \textit{mclogit}} package of \textbf{R},
to model the likelihood of duplicate pull requests being accepted
based on the matched pair samples in our dataset (\ie two paired duplicate pull requests in which one is merged and the other is rejected).
{\color{hltext}
Similar to the analysis in Section~\ref{ss:differenceanalysis},
we add one level metrics at a time and build a model
instead of building one model with all metrics at once.
As a result,
we compare the fit of three models:
a)~\textit{Model 1}, which includes only the patch-level variables,
b)~\textit{Model 2}, which adds submitter-level variables,
a)~\textit{Model 1}, which includes only the submitter-level variables,
b)~\textit{Model 2}, which adds patch-level variables,
and c)~\textit{Model 3}, which adds review-level variables.
The variable transformation and multicollinearity control in the model construction process is similar to those in Section~\ref{ss:differenceanalysis}.
The variable processing phase left us with \hl{***} metrics,
The variable processing phase left us with 19 metrics,
which can be seen in Table~\ref{tab:sub_metrics}.
}
@ -579,9 +581,9 @@ which can be seen in Table~\ref{tab:sub_metrics}.
\subsubsection{Analysis results}
Overall,
as shown in Table~\ref{tab:sta_models},
\hl{both Model 2 and Model 3 have achieved remarkable performance
given their high value of} AUC~\cite{metz1978basic}.
Since Model 3 performs better than Model 2 (AUC: 0.871 vs 0.845) and
both Model 2 and Model 3 have achieved remarkable performance
given their high value of AUC~\cite{metz1978basic}.
Since Model 3 performs better than Model 2 (AUC: 0.890 vs 0.856) and
they have obtained consistent variable effects (\ie there is no significant effect flipping from positive to negative and vice versa),
we discuss their effects based on Model 3.
@ -603,33 +605,59 @@ we discuss their effects based on Model 3.
& Coeffs. & Errors & Signif. & Coeffs. & Errors & Signif.& Coeffs. & Errors & Signif.\\
\midrule
\texttt{log(prev\_pullreqs + 0.5)} & -0.076 & 0.043 & & -0.088 & 0.049 & & -0.083 & 0.052 & \\
\texttt{log(prev\_prs\_acc + 0.5)} & 1.567 & 0.201 & *** & 1.687 & 0.232 & *** & 1.573 & 0.245 & *** \\
\texttt{log(followers + 0.5)} & 0.059 & 0.036 & & 0.061 & 0.042 & & 0.062 & 0.045 & \\
\texttt{first\_pr\_proj TRUE} & -0.217 & 0.162 & & -0.166 & 0.184 & & -0.171 & 0.198 & \\
\texttt{core\_team TRUE} & 0.262 & 0.140 & & 0.202 & 0.164 & & 0.264 & 0.179 & \\
\texttt{log(social\_strength + 0.5)} & 0.387 & 0.148 & ** & 0.238 & 0.167 & & 0.291 & 0.183 & \\
\midrule
\texttt{early\_arrival TRUE} & - & - & - & 0.522 & 0.077 & *** & 0.476 & 0.086 & *** \\
\texttt{log(loc + 0.5)} & - & - & - & 0.418 & 0.081 & *** & 0.372 & 0.085 & *** \\
\texttt{log(files + 0.5)} & - & - & - & 0.217 & 0.160 & & 0.073 & 0.167 & \\
\texttt{test\_inclusion TRUE} & - & - & - & 0.278 & 0.134 & * & 0.283 & 0.243 & \\
\texttt{issue\_tag TRUE} & - & - & - & 0.321 & 0.226 & & 0.277 & 0.148 & \\
\texttt{log(text\_len + 0.5)} & - & - & - & 0.265 & 0.136 & & 0.136 & 0.069 & * \\
\midrule
\texttt{log(revision + 0.5)} & - & - & - & 0.155 & 0.066 & * & 0.284 & 0.106 & ** \\
\texttt{log(comments + 0.5)} & - & - & - & - & - & - & 0.051 & 0.087 & \\
\texttt{log(comments\_inline + 0.5)} & - & - & - & - & - & - & 0.216 & 0.097 & * \\
\texttt{comments\_pos>comments\_neg TRUE} & - & - & - & - & - & - & 0.442 & 0.138 & ** \\
\texttt{CI Success} & - & - & - & - & - & - & 0.726 & 0.202 & *** \\
\texttt{CI Pending} & - & - & - & - & - & - & 1.184 & 0.224 & *** \\
\texttt{CLA Success} & - & - & - & - & - & - & 1.559 & 2.992 & \\
\texttt{CLA Pending} & - & - & - & - & - & - & 2.932 & 2.989 & \\
\texttt{CR Success} & - & - & - & - & - & - & 1.7e+01 & 2.5e+03 & \\
\texttt{CR Pending} & - & - & - & - & - & - & 1.7e+01 & 2.5e+03 & \\
% \texttt{log(prev\_pullreqs + 0.5)} & -0.076 & 0.043 & & -0.088 & 0.049 & & -0.083 & 0.052 & \\
% \texttt{log(prev\_prs\_acc + 0.5)} & 1.567 & 0.201 & *** & 1.687 & 0.232 & *** & 1.573 & 0.245 & *** \\
% \texttt{log(followers + 0.5)} & 0.059 & 0.036 & & 0.061 & 0.042 & & 0.062 & 0.045 & \\
% \texttt{first\_pr\_proj TRUE} & -0.217 & 0.162 & & -0.166 & 0.184 & & -0.171 & 0.198 & \\
% \texttt{core\_team TRUE} & 0.262 & 0.140 & & 0.202 & 0.164 & & 0.264 & 0.179 & \\
% \texttt{log(social\_strength + 0.5)} & 0.387 & 0.148 & ** & 0.238 & 0.167 & & 0.291 & 0.183 & \\
% \midrule
% \texttt{early\_arrival TRUE} & - & - & - & 0.522 & 0.077 & *** & 0.476 & 0.086 & *** \\
% \texttt{log(loc + 0.5)} & - & - & - & 0.418 & 0.081 & *** & 0.372 & 0.085 & *** \\
% \texttt{log(files + 0.5)} & - & - & - & 0.217 & 0.160 & & 0.073 & 0.167 & \\
% \texttt{test\_inclusion TRUE} & - & - & - & 0.278 & 0.134 & * & 0.283 & 0.243 & \\
% \texttt{issue\_tag TRUE} & - & - & - & 0.321 & 0.226 & & 0.277 & 0.148 & \\
% \texttt{log(text\_len + 0.5)} & - & - & - & 0.265 & 0.136 & & 0.136 & 0.069 & * \\
% \midrule
% \texttt{log(revision + 0.5)} & - & - & - & 0.155 & 0.066 & * & 0.284 & 0.106 & ** \\
% \texttt{log(comments + 0.5)} & - & - & - & - & - & - & 0.051 & 0.087 & \\
% \texttt{log(comments\_inline + 0.5)} & - & - & - & - & - & - & 0.216 & 0.097 & * \\
% \texttt{comments\_pos>comments\_neg TRUE} & - & - & - & - & - & - & 0.442 & 0.138 & ** \\
% \texttt{CI Success} & - & - & - & - & - & - & 0.726 & 0.202 & *** \\
% \texttt{CI Pending} & - & - & - & - & - & - & 1.184 & 0.224 & *** \\
% \texttt{CLA Success} & - & - & - & - & - & - & 1.559 & 2.992 & \\
% \texttt{CLA Pending} & - & - & - & - & - & - & 2.932 & 2.989 & \\
% \texttt{CR Success} & - & - & - & - & - & - & 1.7e+01 & 2.5e+03 & \\
% \texttt{CR Pending} & - & - & - & - & - & - & 1.7e+01 & 2.5e+03 & \\
\texttt{log(prev\_pullreqs + 0.5)} & -0.078 & -1.832 & & -0.097 & -1.987 & * & -0.091 & -1.738 & \\
\texttt{log(prev\_prs\_acc + 0.5)} & 1.597 & 7.884 & *** & 1.729 & 7.363 & *** & 1.644 & 6.617 & *** \\
\texttt{log(followers + 0.5)} & 0.061 & 1.674 & & 0.062 & 1.477 & & 0.064 & 1.415 & \\
\texttt{first\_pr\_proj TRUE} & -0.206 & -1.270 & & -0.169 & -0.911 & & -0.173 & -0.868 & \\
\texttt{core\_team TRUE} & 0.251 & 1.786 & & 0.184 & 1.111 & & 0.237 & 1.312 & \\
\texttt{log(social\_strength + 0.5)} & 0.383 & 2.590 & ** & 0.217 & 1.295 & & 0.283 & 1.538 & \\
\midrule
\texttt{early\_arrival TRUE} & - & - & - & 0.534 & 6.865 & *** & 0.483 & 5.572 & *** \\
\texttt{log(loc + 0.5)} & - & - & - & 0.462 & 5.701 & *** & 0.386 & 4.477 & *** \\
\texttt{log(files + 0.5)} & - & - & - & 0.388 & 2.341 & * & 0.200 & 1.135 & \\
\texttt{test\_inclusion TRUE} & - & - & - & 0.239 & 1.034 & & 0.200 & 0.805 & \\
\texttt{issue\_tag TRUE} & - & - & - & 0.255 & 1.855 & & 0.261 & 1.734 & \\
\texttt{log(text\_len + 0.5)} & - & - & - & 0.152 & 2.253 & * & 0.133 & 1.877 & \\
\midrule
\texttt{log(revision + 0.5)} & - & - & - & - & - & - & 0.275 & 2.563 & * \\
\texttt{log(comments + 0.5)} & - & - & - & - & - & - & 0.036 & 0.406 & \\
\texttt{log(comments\_inline + 0.5)} & - & - & - & - & - & - & 0.208 & 2.138 & * \\
\texttt{comments\_pos>comments\_neg TRUE} & - & - & - & - & - & - & 0.460 & 3.315 & *** \\
\texttt{CI Success} & - & - & - & - & - & - & 0.728 & 3.574 & *** \\
\texttt{CI Pending} & - & - & - & - & - & - & 1.175 & 5.226 & *** \\
\texttt{CLA Success} & - & - & - & - & - & - & 1.596 & 0.532 & \\
\texttt{CLA Pending} & - & - & - & - & - & - & 2.948 & 0.984 & \\
\texttt{CR Success} & - & - & - & - & - & - & 1.4e+01 & 0.022 & \\
\texttt{CR Pending} & - & - & - & - & - & - & 1.4e+01 & 0.022 & \\
\midrule
Area Under the ROC Curve &\multicolumn{3}{c}{0.704}& \multicolumn{3}{c}{0.853} & \multicolumn{3}{c}{0.888}\\
Area Under the ROC Curve &\multicolumn{3}{c}{0.704}& \multicolumn{3}{c}{0.856} & \multicolumn{3}{c}{0.890}\\
\bottomrule
@ -646,48 +674,57 @@ we discuss their effects based on Model 3.
{\color{hltext}
As for the submitter-level predictors,
\texttt{prev\_pullreqs}, \texttt{prev\_prs\_acc},
and \texttt{prior\_interaction} have significant effects.
Interestingly,
the two experience-related metrics present opposite effects,
\ie, \texttt{prev\_pullreqs} have a negative effect and \texttt{prev\_prs\_acc} have a positive effect.
only the predictor \texttt{prev\_prs\_acc}
has a significant, positive effect.
This means that
duplicate pull requests submitted by
developers with a higher number of previous pull requests
are less likely to be accepted by integrators,
however if the developers' previous pull requests have a higher acceptance rate
their duplicate pull requests are more likely to be accepted.
\hl{****talk****}
The predictor \texttt{prior\_interaction} has a strong, positive association with the acceptance of duplicate pull requests.
This reveals that duplicate pull requests from developers having a stronger social connection to the project have a higher chance to be accepted.
duplicate pull requests submitted by experienced developers
whose previous pull requests have a higher acceptance rate
are more likely to be accepted.
This is inline with the prior findings about general pull request evaluation~\cite{gousios2015work,zou2019how,Kononenko2018Studying,jiang2013will}.
Perhaps surprisingly,
the predictors \texttt{core\_team} and \texttt{followers} are not significant.
the predictors relating to developers' standing ( \texttt{core\_team} and \texttt{followers}) and
their social connection to the project (\texttt{social\_strength}) are not significant.
Prior studies have shown that
pull requests from the developers holding higher status in the community and a specific project have higher acceptance rates~\cite{Tsay2014Influence,yu16det}.
pull requests from the developers holding higher standing and having stronger social connection with the project have higher acceptance rates~\cite{Tsay2014Influence,yu16det}.
However,
our model shows that
developers' high status do not increase the likelihood of duplicate pull request acceptance.
\hl{***more talk***}
our model shows that these factors do not increase the likelihood of duplicate pull request acceptance.
We present an assumption that
in the context of selection between duplicates,
integrators do not make much use of social signals when making decisions
in order to ensure fairness within the community~\cite{Jensen2007Role, baysal2012secret}.
}
For patch-level metrics,
\texttt{early\_arrival} and \texttt{log} are highly significant in the model.
\texttt{early\_arrival} and \texttt{loc} are highly significant in the model.
As expected,
\textbf{
duplicate pull requests submitted earlier have a higher likelihood of being accepted.}
We can also observe that the odds of being accepted increase by a factor of 3.78($e^{1.233}$)
for every $e$ factor increase in the number of LOC (\textit{n\_LOCs}),
which indicates that \textbf{duplicate pull requests changing more LOCs are more likely to be accepted}.
% \textbf{
duplicate pull requests submitted earlier have a higher likelihood of being accepted.
% }
We can also observe that the odds of being accepted increase by a factor of 1.63 ($e^{0.487}$)
for every $e$ factor increase in the number of \texttt{loc},
which indicates that
% \textbf{
duplicate pull requests changing more locs are more likely to be accepted.
% }
This finding may have several explanations.
First,
large pull requests might address more issues than small ones;
therefore, accepting large pull requests seeks like a ``good deal''
the larger pull request might address more issues than smaller one;
therefore,
accepting the larger one like a ``good deal''
since it can kill two birds with one stone.
Second,
more changed LOCs might means that more work have been done in assuring the high standard and robustness of the change,
such as necessary handling of possible exceptions and updates on the test files and change log.
\hl{*** explanation needs evidence}
more changed locs might mean that more work have been done to achieve a more thorough solution, \eg cleaning up unused code.
{\color{hltext}
Although integrators do not like large pull requests composed of loosely related changes,
they may not mind accepting a large pull request which is coherent and with a clear purpose~\cite{Kononenko2018Studying}.
% We also see that the predictor \texttt{text\_len} present a positive effect,
% which indicates that duplicate pull requests with longer description have higher chances of acceptance.
% We conjecture that it might be because longer description provide better documentation of the pull request
% which can facilitate the evaluation process~\cite{yu16det}.
}
Finally,
@ -695,23 +732,37 @@ we discuss the review-level metrics.
The predictor \texttt{comments\_inline}
has a significant, positive effect.
This indicates that
\textbf{duplicate pull requests receiving more inline comments have a higher chance of being accepted}.
% \textbf{
duplicate pull requests receiving more inline comments have a higher chance of being accepted.
% }
Receiving more comments, especially inline comments,
means that a pull request has been reviewed and discussed by integrators with higher effort investment.
Consequently,
integrators tend to reject the duplicate pull requests with less comments,
otherwise they need to invest additional review effort.
might mean that a pull request has been reviewed and discussed more thoroughly to some extent.
Therefore,
integrators are more willing to prefer the highly discussed one that has a low probability of containing potential defects.
% means that a pull request has been thoroughly reviewed and discussed by integrators with higher effort investment.
% Consequently,
% integrators tend to reject the duplicate pull requests shallowly reviewed with less comments,
% otherwise they need to invest additional review effort.
% }
As for comment sentiment,
we can see that
\textbf{duplicate pull requests receiving more positive comments than negative comments are more likely to be accepted}.
\hl{***more talk***}
unsurprisingly,
% \textbf{
duplicate pull requests receiving more positive comments than negative comments are more likely to be accepted.
% }
In terms of DevOps checking,
as expected,
the predictor \texttt{BT} has a positive effect
whenever its value is \texttt{success} or \texttt{pending}
when its value is \texttt{success} or \texttt{pending}
compared to when the value is \texttt{failure}.
This means that \textbf{duplicate pull requests have lower likelihood of being accepted
if the CI test result is failure}.
\hl{***more talk***}
This means that
% \textbf{
duplicate pull requests have lower likelihood of being accepted
if the CI test result is failure.
{\color{hltext}
This confirms the finding in the previous study~\cite{yu16det} that
CI failure has a significant, negative effect on pull request acceptance.
}
% }
\subsection{Manual inspection.}
@ -733,9 +784,11 @@ This process includes the following three steps.
\vspace{0.5em}
\noindent\textbf{Card preparation: }
For each randomly selected duplicate pair,
we examine their review history and extract the comments made by integrators
to explain why they prefer one duplicate over the other one.
The extracted comments are then recorded in a card.
{\color{hltext}
we read the dialogue and select all the comments expressing integrators' choice preference between duplicates.
}
% and extract the comments made by integrators to explain why they would prefer one duplicate over the other one.
The selected comments are then recorded in a card.
\vspace{0.5em}
\noindent\textbf{Pair execution: }
@ -791,7 +844,7 @@ we discuss each of reasons supported by examples.
& Higher quality &19 &\tblct{1.33}{72}{19}\\
& Correctness &11 &\tblct{1.33}{72}{11}\\
& Broader coverage &7 &\tblct{1.33}{72}{7}\\
& Inclusion of test codes &5 &\tblct{1.33}{72}{5}\\
& Inclusion of test code &5 &\tblct{1.33}{72}{5}\\
& Pushed to proper branch &4 &\tblct{1.33}{72}{4}\\
\midrule
@ -877,6 +930,7 @@ the existence of test codes in the duplicate pull requests could
help win some favor from integrators (
\textit{e.g.},
\textit{``Thanks for the fix. Not sure we want to merge this or wait for \#3907 which also fixes it and adds a regression test for all estimators''} ).
\hl{\texttt{test\_inclusion} is not signifant}
@ -998,7 +1052,7 @@ For example, although \textit{rails/rails/\#20697} was preferred over \textit{ra
the author of \textit{rails/rails/\#20697} was asked to incorporate the tests from \textit{rails/rails/\#20050}
(\textit{``I prefer this behavior (and its simpler implementation) Can you please cherry-pick the test from \#20050''}).
In such cases,
integrators usually also gave credits to the authors of the rejected duplicates for the incorporated codes
integrators usually also gave credits to the author of the rejected pull request for the incorporated codes
(\textit{``along with adding an changelog entry listing both authors'' and ``Can you credit him in the commit as well after you squash and rebase''}).
@ -1018,13 +1072,14 @@ as the integrator said:
\vspace{0.5em}
\noindent\textbf{\hl{Mutul-assessment}.}
Integrators might ask the authors of the rejected duplicates to evaluate the preferred ones
Integrators might ask the author of the rejected duplicate pull request to evaluate the preferred
(\eg \textit{``Could you jump on \#3082 and help us review it?''}
and \textit{``Please work with @xxx to test his code once he fixes the merge conflicts''}).
We also found that some of them were willing to help review the counterparts,
as two authors said:
\textit{``Made a minor note on the other PR to include one of the improvements} and
We also found that some of the authors were willing to offer help
(\eg
\textit{``Made a minor note on the other PR to include one of the improvements''} and
\textit{``Looks like they took my advice and used proc\_terminate() so another package was not needed. Thanks for catching duplicate, but at least it was not a waste. :)''}
)
}
@ -1033,13 +1088,13 @@ as two authors said:
\textit{
\textbf{RQ3:}
Pull requests with
accurate and high-quality implementation, broad coverage, necessary test codes, and deep discussion
accurate and high-quality implementation, broad coverage, necessary test code, high maturity, and deep discussion
are more likely to be accepted.
However,
integrators also make a choice based on non-technical considerations,
e.g., they may accept pull requests in order to encourage newcomers or respect the arrival order and active response.
\hl{For the rejected duplicates,
For the rejected duplicates,
integrators might try to maximize their value,
e.g., incorporating the useful codes.}
e.g., incorporating the useful codes.
}
\end{mdframed}

203
5_discussion.tex
View File

@ -23,7 +23,9 @@ and made them transparent and visible~\cite{Dabbish2012Social},
which help developers to maintain awareness with less effort~\cite{Kalliamvakou2015Open}.
However,
awareness breakdown still occurs and results in duplicate work.
Our analysis of the specific context where duplicates occur reveal
Our analysis of the specific context where duplicates are produced,
as shown in Section~\ref{ss:contextanalysis},
reveal
three mismatches leading to awareness breakdown.
@ -37,18 +39,19 @@ whenever a developer work on an issue
because other developers might self-assign themselves to the same issue.
However,
our findings show that
some contributors lack sufficient effort investment in awareness activities (contexts \textbf{Not searching for existing work}, \textbf{Overlooking linked pull requests}, and \textbf{Overlooking existing claim} ).
some contributors lack sufficient effort investment in awareness activities
({\footnotesize Section~\ref{ss:contextanalysis}: \textit{\textbf{Not searching for existing work}}, \textit{\textbf{Overlooking linked pull requests}}, and \textit{\textbf{Overlooking existing claim}}}).
We assume that this is due to the volunteer nature of OSS participation.
For some developers,
especially the casual contributors and one time contributors,
a major motivation to make contributions is to \textit{``scratch their own itch''}~\cite{pinto2016more,lee2017understanding}.
When they encounter a problem,
they code a patch to fix it and send the patch back to the community.
Some of them even dot care about the final outcome of their pull requests~\cite{steinmacher2018almost}.
Some of them even do not care about the final outcome of their pull requests~\cite{steinmacher2018almost}.
It might be harder to get them spend more time to maintain awareness
of other developers.
Automatic awareness tools can mitigate this problem.
Prior has proposed to automatically detect duplicates at pull request submission~\cite{li2017detecting,ren2019identifying} and identify ongoing features from forks~\cite{zhou2018identifying}.
Prior research has proposed to automatically detect duplicates at pull request submission~\cite{li2017detecting,ren2019identifying} and identify ongoing features from forks~\cite{zhou2018identifying}.
Furthermore,
we advocate for future research on seamlessly integrating awareness tools to developers' development environment
and designing intelligent and non-intrusive notification mechanism.
@ -65,7 +68,8 @@ GitHub provides developers with a wide range of mechanisms,
to maintain a general awareness about project status.
% in terms of historical, current, and even upcoming events.
However,
developers can be overwhelmed with a large-scale of incoming events in popular projects (context \textbf{Missing notifications}).
developers can be overwhelmed with a large-scale of incoming events in popular projects
({\footnotesize Section~\ref{ss:contextanalysis}: \textit{\textbf{Missing notifications}}}).
It is also impractical for developers to always maintain overall awareness of a project
due to multitasking~\cite{vasilescu2016sky} and turnover~\cite{Lin2017Developer}.
Usually,
@ -77,21 +81,28 @@ the main mechanisms to meet this demand
are querying issue and pull request list and reading through the discussion history.
As mentioned in Section~\ref{ss:contextanalysis},
the support by these mechanisms is not as adequate as expected
due to information scattering and mixture (contexts \textbf{Overlooking linked pull requests} and \textbf{Overlooking existing claims})
and other technical problems (contexts \textbf{Disappointing search functionality} and \textbf{Diversity of natural language usages}).
due to information mixture
({\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Overlooking linked pull requests}} and \textbf{\textit{Overlooking existing claims}}})
and other technical problems
({\footnotesize Section~\ref{ss:contextanalysis}:
\textbf{\textit{Disappointing search functionality}} and \textbf{\textit{Diversity of natural language usages}}}).
Awareness mechanisms would be most useful
if they can fulfil developers' actual demands in maintaining awareness.
\vspace{0.5em}
\noindent \textbf{A mismatch between awareness *** actual information exchange.}
Maintaining awareness is dual.
\noindent \textbf{A mismatch between awareness maintenance and actual information exchange.}
Maintaining awareness is
% dual.
bidirectional.
Intuitively,
it means developers need to \textit{gather external information} to stay aware of others' activities.
it means that developers need to \textit{gather external information} to stay aware of others' activities.
But from a global perspective,
it also means developers should actively \textit{share their personal information} that can be gathered by others.
Our findings show that some developers do not timely announce their plans (context \textbf{Implementing without claiming first})
and link their work to the associated issue (context \textbf{Lack of links}).
Our findings show that some developers do not timely announce their plans
({\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Implementing without claiming first}}})
and link their work to the associated issue
({\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Lack of links}}}).
This hinders other developers' ability to gather adequate contextual information.
Although prior work~\cite{Treude2010Awareness,Arora2016Supporting,Calefato2012Social} has extensively studied on
how to help developers track work and get information,
@ -113,7 +124,7 @@ help developers ensure that their individual contributions do not cause accident
\subsubsection{The paradox of duplicates}
Generally speaking,
both project integrators and contributors hope to prevent duplicated pull requests,
both project integrators and contributors hope to prevent duplicate pull requests,
because duplicates can waste their time and effort,
as shown in Section~\ref{rq1}.
This is also reflected in their comments,
@ -122,10 +133,10 @@ This is also reflected in their comments,
\textit{``No need to do the same thing in two PRs''},
and \textit{``Oops! Sorry, did not mean to double up''}.
However,
when duplicates already exist,
some positive value might be mined from them
except for the negative impacts,
as shown in Section~\ref{beyond}
when duplicates are already produced,
some potential value might be mined from them
% except for the negative impacts,
as shown in Section~\ref{beyond}.
% 在管理者看来
\vspace{0.5em}
@ -138,7 +149,7 @@ duplicates implemented in different approaches
provide alternative solutions,
as a developer put it:
\textit{``The pull requests are different, so maybe it is good there are two.''}
As a result,
In such cases,
project integrators have a higher chance to accept a better patch.
However,
this comes at a price.
@ -161,15 +172,17 @@ one contributor tried to persuade integrators to choose his pull request rather
\textit{``Pick me! Pick me! I was first! :)''}.
Nevertheless,
we found some cases where
authors of duplicates worked together towards a more appropriate patch by means of mutual assessment and
the authors of duplicates worked together towards a better patch by means of mutual assessment and
negotiated incorporation,
as shown in Section~\ref{beyond}.
The collaboration not only leads to a better solution
but also allows contributors to learn from each other's strength (\textit{`` I looked at some awesome code that xxx wrote to fix this issue and it was so simple, I just did not fully understand the issue I was fixing.''})
and fosters a friendly collaborative environment.
According to developers,
the collaboration is also an opportunity for both the authors
to learn from each other's strength
(\textit{`` I looked at some awesome code that xxx wrote to fix this issue and it was so simple, I just did not fully understand the issue I was fixing.''}).
% and fosters a friendly collaborative environment.
Standing for their own patches,
but seeking collaboration,
is a trade-off authors of duplicates should be aware of.
but seeking for collaboration and learning,
is a trade-off the authors of duplicates should be aware of.
@ -195,26 +208,29 @@ what were observed in the context of general pull request evaluation.
\vspace{0.5em}
\noindent \textbf{Community fairness matters:}
Some of our findings confirm the results of previous studies that
pull requests with accurate and high-quality implementation (predictor \texttt{CI} and reasons \textbf{Higher quality} and \textbf{Correctness}) and submitted by experienced developers (predictor \texttt{prev\_prs\_acc}) are more likely to be accepted~\cite{yu16det,gousios2015work,zou2019how,Kononenko2018Studying}.
Our findings confirm the results of previous studies that
pull requests with accurate and high-quality implementation and submitted by experienced developers
are more likely to be accepted~\cite{yu16det,gousios2015work,zou2019how,Kononenko2018Studying}.
However,
social metrics,
\eg the submitter's status (predictors \texttt{followers} and \texttt{core\_team})
and the social connection between the submitter with integrators (predictor \texttt{social\_strength}),
do not have as strong effects on the acceptance of duplicates pull request as observed in prior work~\cite{Tsay2014Influence,yu16det}.
the submitter's standing and the social connection between the submitter with integrators,
do not have as strong effects on the acceptance of duplicate pull request as observed in prior work~\cite{Tsay2014Influence,yu16det}.
Instead,
we observe that a more objective factor, \ie the arrival order of duplicates,
presents a strong influence.
Both our regression model (predictor \texttt{early\_arrival}) and manual observation (reason \textbf{First-come, First-served}) show that
duplicate pull requests arriving earlier are more likely to be accepted.
presents a strong effect.
Both our regression model
% (predictor \texttt{early\_arrival})
and manual observation
% (reason \textbf{First-come, First-served})
show that
duplicate pull requests arriving earlier than their counterparts are more likely to be accepted.
% This factor belongs to neither social metric nor technical metric.
Respecting the arrival order is like a default community rule
developers usually follow to manage duplicates~\cite{sun2011towards,sun2010discriminative,stackexchange-dup}.
% A similar treatment \textit{``first-in, first-out''} was also observed
% when integrators prioritize their work on pull request evaluation~\cite{gousios2015work}.
It can reveal that integrators focus more the technical metrics and objective difference
when making decision between duplicates
in order to ensure fairness within the community.
It might reveal that integrators focus more on the technical metrics and objective difference
when making decisions between duplicates
in order to ensure fairness within the community~\cite{Jensen2007Role, baysal2012secret}.
% explaining rejection big challenge
@ -223,23 +239,26 @@ in order to ensure fairness within the community.
% revision; in_line comments
While prior work on pull request acceptance has suggested that highly discussed pull requests are less likely to be accepted~\cite{Tsay2014Influence}.
Our model shows that
duplicates pull requests with more inline comments (predictor \texttt{comments\_inline})
and revisions (predictor \texttt{revisions})
duplicate pull requests with more inline comments
% (predictor \texttt{comments\_inline})
and revisions
% (predictor \texttt{revisions})
have higher chances of acceptance.
A higher number of review comments and revisions might indicate that
the pull request is not perfect and integrators request changes.
the pull request is not perfect and integrators request changes to improve it.
However,
it also reflects that the pull request has been thoroughly reviewed,
both integrators and the submitter have invested considerable effort.
and both integrators and the submitter have invested considerable effort.
If the pull request does not have fatal drawbacks and the other duplicate pull request do not provide significant benefits,
integrators should prefer the thoroughly reviewed pull request other than the shallowly reviewed one.
integrators should prefer the thoroughly reviewed pull request rather than the shallowly reviewed one.
After all,
the top challenge faced by integrators is time~\cite{gousios2015work}, and therefore
they should avoid putting more redundant effort on the shallowly reviewed duplicates.
the top challenge faced by integrators is time~\cite{gousios2015work},
and it is reasonable for them to avoid putting more redundant effort on the shallowly reviewed duplicates.
Moreover,
asking for more work from developers to improve their pull requests might be difficult~\cite{gousios2015work,steinmacher2018almost}.
asking for more work from contributors to improve their pull requests might be difficult~\cite{gousios2015work,steinmacher2018almost}.
Thus,
choosing the thoroughly discussed duplicate pull requests of high maturity (\ie higher \texttt{revisions})
choosing the thoroughly discussed duplicate pull requests of high maturity
% (\ie higher \texttt{revisions})
might be a safe decision.
@ -258,11 +277,12 @@ when they are involved in the pull-based development model.
{\color{hltext}
Many duplicates were produced because
contributors did not conduct adequate checking to
make sure that no one else is working on the same thing (contexts \textbf{Not searching for existing work}, \textbf{Overlooking linked pull requests}, and \textbf{Overlooking existing claims}).
make sure that no one else is working on the same thing ({\footnotesize Section~\ref{ss:contextanalysis}:
\textbf{\textit{Not searching for existing work}}, \textbf{\textit{Overlooking linked pull requests}}, and \textbf{\textit{Overlooking existing claims}}}).
We recommend that
contributors should perform at least three kinds of checking:
\textit{i)} reading through the whole discussion of an issue and checking whether anyone has claimed the issue,
\textit{ii)} examining each of the pull requests linked to an issue and checking whether any of them is an ongoing work to solve the issue, and
\textit{i)} reading through the whole discussion of an issue and checking whether anyone has claimed the issue;
\textit{ii)} examining each of the pull requests linked to an issue and checking whether any of them is an ongoing work to solve the issue; and
\textit{iii)} performing searches with different keywords against open and closed pull requests and issues,
and carefully checking where similar work already exists.
}
@ -277,20 +297,32 @@ it might be difficult for them to complete an individual task in a timely fashio
However,
we still suggest that contributors should quickly accomplish each work in proper order, \eg one item at a time, to shorten their local duration.
This can make their work publicly visible earlier,
which can, to some extent, prevent others from submitting duplicates ({\color{hltext}context \textbf{Overlong local work}}).
which can, to some extent, prevent others from submitting duplicates
({\color{hltext}
{\footnotesize Section~\ref{ss:contextanalysis}:
\textbf{\textit{Overlong local work}}
}}).
\vspace{0.5em}
\noindent \textbf{Precise context:}
Providing complete and clear textual information for submitted pull requests is helpful
for other contributors to retrieve these pull requests and acquire an accurate and comprehensive understanding of them
({\color{hltext} context \textbf{Diversity of natural language usage}}).
({\color{hltext}
{\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Diversity of natural language usage}}
}}).
In addition,
if a pull request is solving a tracked issue,
adding the issue reference in the pull request description,
\textit{e.g., ``fix \#[issue\_number]'',}
% helps to aggregate related activities concerning the same issue.
can avoid some duplicates because
other developers would have a chance to find the linked pull request ({\color{hltext} context \textbf{Lack of links}}).
{\color{hltext}
can avoid some duplicates because of
the increasing degree of awareness.
%other developers would have a chance to find the linked pull request
( {\footnotesize Section~\ref{ss:contextanalysis}:
\textbf{\textit{Lack of links}}
}).
}
\vspace{0.5em}
@ -298,7 +330,10 @@ other developers would have a chance to find the linked pull request ({\color{hl
{\color{hltext}
Zhou \etal~\cite{zhou2019fork} already suggested that
claiming an issue upfront is associated with a lower chance of redundant work.
We would like to emphasize again the importance of early declaration base our findings ({\color{hltext}\textbf{Implementing without claiming first}}).
We would like to emphasize again the importance of early declaration base our findings ({\color{hltext}
{\footnotesize Section~\ref{ss:contextanalysis}:
\textbf{\textit{Implementing without claiming first}}
}}).
If contributors decide to submit a pull request to an issue,
they are better to first declare their intentions before coding,
instead of reporting their patches after the work is done.
@ -323,7 +358,8 @@ Authors of duplicates can review each of other's patch and discuss the differenc
before waiting for an official statement from the core team.
This can provide a solid basis for integrators to make informed decisions about which duplicate should be accepted.
Moreover,
even a duplicate is finally closed,
if the value of a duplicate pull request has been explicitly stated,
even it is finally closed,
its useful part has a higher chance to be noticed and cherry-picked by integrators,
as shown in Section~\ref{beyond}.
}
@ -347,7 +383,9 @@ usually have a contribution guideline
telling contributors how to report an issue, submit a pull request and \etc
We found many of them have warned contributors not to submit duplicate issues and pull requests.
Since
many developers said they did not or forgot to check for existing work (contexts \textbf{Not searching for existing work}, \textbf{Overlooking Overlooking linked pull requests}, and \textbf{Overlooking existing claims}).
many developers said they did not or forgot to check for existing work
({\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Not searching for existing work}}, \textbf{\textit{Overlooking linked pull requests}}, and \textbf{\textit{Overlooking existing claims}}
}).
projects can make the warning more visible (\eg highlighted in bold) and
easy-follow (\eg clearly itemizing the behaviors should be taken to check or existing pull requests).
}
@ -360,21 +398,17 @@ easy-follow (\eg clearly itemizing the behaviors should be taken to check or exi
Integrators must make a choice between duplicate pull requests,
which means that they have to reject someone.
For contributors whose pull requests have been rejected,
it is reasonable to provide feedback and explanation about
why their work has been rejected rather than simply closing their pull requests.
After all,
they have devoted time and energy to their contribution.
they might be pleased to get feedback and explanation about why their work has been rejected rather than simply closing their pull requests.
% After all,
% they have devoted time and energy to their contribution.
{\color{hltext}
However,
we observed many cases where decisions were made without detailed explanation ({\color{hltext} reason \textbf{No explanation} in Table~\ref{tab:reason}}).
we observed nearly 50\% of our qualitative samples where decisions were made without any explanation (as shown in Table~\ref{tab:reason}).
Even worse,
we identified a case where an inaccurate explanation for the decision somewhat displeased the contributor
we identified that the rough explanation (\eg \textit{``Thanks for your PR but this fix is already merged in \#20610''}) would be likely to make the contributor upset
(\textit{``'already' implies I submitted my PR later than that, rather than nearly a year earlier ;) But at least it's fixed.''}).
Prior study~\cite{gousios2016work} show that contributors are fear of rejection and dislike unconstructive attitude.
We conjecture that a rational and polite explanation might eliminate contributors' embarrassing feeling of submitting duplicates
which might weaken their intrinsic motivation for OSS participation.
A detailed explanation for rejection is also helpful for the contributors
to learn how to prepare more satisfactory pull requests later.
In that case, the integrator had to give an additional apology (\textit{``sorry, sometimes a PR falls in the cracks and a newer one gets the attention. We have improved the process in hopes to avoid this but we still have a big backlog in which these things are present.''}) to mitigate the negative effect.
In the future, a careful analysis should be designed to examine the effectiveness of this suggestion based on controlled experiments.
}
@ -409,15 +443,15 @@ In order to make it more efficient for developers to maintain awareness of each
we envision a new mechanism called \textit{Claim} which is described as follows.
For a GitHub issue,
each developer who is interested in it
can click the \textit{Claim} button on the issue tracking page to claim that s/he is going to work on the issue.
can click the \textit{Claim} button on the issue page to claim that s/he is going to work on the issue.
The usernames of all claimers of the issue
are listed together below the \textit{Claim} button.
Every time the \textit{Claim} button is clicked,
an empty pull request is automatically generated
and linked to the claimer's username in the issue claimer list.
Moreover,
claimers have a chance to report their plans about how to fix an issue,
which would be used to describe the empty pull request.
claimers have a chance to report their plans about how to fix the issue in the input box displayed when the \textit{Claim} button is clicked.
The reported plans would be used to describe the empty pull request.
Subsequently,
claimers perform updates of the empty pull request by submitting codes
until they produce a complete patch.
@ -429,7 +463,7 @@ this mechanism makes it more convenient for developers to
share their intentions and activities through just clicking a button.
On the other hand,
developers can efficiently catch and track other developers' intentions and activities
by watching issue claimer list.
by checking the issue claimer list.
}
@ -450,7 +484,9 @@ The features discussed in Section~\ref{ss:metrics} can also be integrated to enh
{\color{hltext}
\vspace{0.5em}
\noindent \textbf{Reference reminder:}
Since developers might overlook linked pull requests to issues ({\color{hltext} context \textbf{Overlooking linked pull requests}}),
Since developers might overlook linked pull requests to issues
({\color{hltext}
{\footnotesize Section~\ref{ss:contextanalysis}: \textbf{\textit{Overlooking linked pull requests}}}}),
platforms can actively remind developers of existing pull requests linked to the same issue at pull request submission time.
The goal of this functionality is similar to that of the duplicate detection tool,
However, it can be implemented in a more straightforward way.
@ -494,16 +530,23 @@ cherry-picking the needed commits from or rebase onto the remote branch,
and
\textit{iv)}
updating $PR_{j}$ by synchronizing the changes from local repository to the head branch of $PR_{i}$.
Sometimes the author of $PR_{j}$ might be asked to
add the author name of $PR_{i}$ in the latest commit message or project changelog to give credit
for the incorporated codes.
This process can be too complex for newcomer developers to undertake.
% Sometimes the author of $PR_{j}$ might be asked to
% add the author name of $PR_{i}$ in the latest commit message or project changelog to give credit
% for the incorporated codes.
Sometimes developers also need to update the commit message or project changelog
to give credit for the incorporated codes.
This process can be too complex for newcomer to undertake.
Moreover,
this process looks tedious for incorporating trivial changes.
Therefore,
GitHub can support online incorporation of pull requests
so that developers can easily pick the needed codes from the displayed diff snippets.
Furthermore,
the incorporated codes can be squashed into a separate commit to make it more effective for credit record.
GitHub can support online incorporation of duplicates pull requests.
For example,
it can allow developers to pick the needed codes by clicking buttons in the UI,
and the credit is given to the picked codes
by automatically updating the commit message and changelog.
% Therefore,
% GitHub can support online incorporation of pull requests
% so that developers can easily pick the needed codes from the displayed diff snippets.
% Furthermore,
% the incorporated codes can be squashed into a separate commit to make it more effective for credit record.
}

40
8_conclusion.tex
View File

@ -12,35 +12,17 @@ and the alternative preference of integrator between duplicate pull requests.
We conducted an empirical study on 26 GitHub projects to achieve the goal.
We found that
duplicate pull requests slow down the review process and require more reviewers for extended discussions.
We observed that duplicate pull requests are significantly different from non-duplicate pull requests in terms of contributors' experience, modification type, modification size, modification hotness, issue visibility, and file type.
We found that duplicate pull requests with accurate and high-quality implementation, broad coverage, necessary test codes, and deep discussion, are more likely to be accepted.
We observed that duplicate pull requests are significantly different from non-duplicate pull requests in terms of project-level characteristics (\eg area hotness and number of active core team members),
submitter-level characteristics (
\eg contribution experience and social connection to project),
and patch-level characteristics (
\eg change type and issue visibility).
% contributors' experience, modification type, modification size, modification hotness, issue visibility, and file type.
We found that duplicate pull requests with accurate and high-quality implementation, broad coverage, necessary test codes, high maturity, and deep discussion, are more likely to be accepted.
We also found that integrators might make a choice based on non-technical considerations,
\textit{e.g.},
they may accept pull requests to encourage newcomers or respect arrival order and active response.
%%%%unbelievable....
%%% In RQ1, we have quantitatively examined
%In analyzing the negative impact of duplicate pull requests, we presented
%the redundant effort that
%duplicate pull requests consume in
%coding, code review, and DevOps checking.
%Moreover, we found that
%duplicate pull requests slow down the review process and require more reviewers for extended discussions.
%%% In RQ2,
%%%% we have investigated the context of duplicate pull requests and found that
%In revealing the context of duplicate pull requests,
%we found that contributors' inappropriate working patterns and the shortcomings of their collaborating environment
%may result in duplicate pull requests.
%Moreover, we observed that duplicate pull requests are significantly different from non-duplicate pull requests in terms of contributors' experience, modification type, modification size, modification hotness, issue visibility, and file type.
%%% In RQ3,
%%% we have investigated the integrators' choice between duplicate pull requests and found that
%In investigating integrators' choice between duplicate pull requests,
%we found that duplicate pull requests with accurate and high-quality implementation, broad coverage, necessary test codes, and deep discussion, are more likely to be accepted.
%In addition, we found that integrators might make a choice based on non-technical considerations,
%\textit{e.g.},
%they may accept pull requests to encourage newcomers or respect arrival order and active response.
Based on the findings
%%%we also propose some practical suggestions.
@ -49,16 +31,16 @@ we recommend that
OSS contributors should always perform sufficient verification against
existing work before they start working on a task.
%Moreover,
Contributors are expected to reveal their intentions as soon as possible
Contributors are expected to declare their intentions as soon as possible
and prepare their work with complete related information to
make their work highly visible early on.
Intergrators should provide detailed explanation and constructive suggestions
when they reject duplicate pull requests,
which helps form a friendly environment and retain contributors.
Social coding platforms are expected to enhance
the integration and monitor of the ecosystem dynamics of a project,
which is helpful to achieve early awareness of developers' activities.
It is also meaningful to provide practical service and tools to support
the awareness mechanisms
in order to make it more effective and efficient for developers to stay aware of each other.
It is also meaningful to provide practical service and tools to support
automatic identification of duplicates,
visualized comparison between duplicates, \etc.

6
IEEEtran.tex
View File

@ -56,7 +56,7 @@
\usepackage{tabularx}
\usepackage{arydshln}
\definecolor{hltext}{rgb}{0.9,0,0}
\definecolor{hltext}{rgb}{0,0,0}
\newcommand{\gh}{GitHub}
\newcommand{\ie}{{\textit{i.e.}},\xspace}
\newcommand{\eg}{{\textit{e.g.}},\xspace}
@ -74,7 +74,9 @@
\begin{document}
\title{An Empirical Study of Duplicate Pull Requests in OSS Projects}
\title{
Redundancy, Context, and Preference:
An Empirical Study of Duplicate Pull Requests in OSS Projects}
\author{
Zhixing Li,

30
experiment_code/major_revision/rq2/contex.py
View File

@ -109,19 +109,25 @@ def omit_problem_pair(DUP_PRS, NOTE, MST_PR, PRJ_ID):
return pairs
def load_pairs():
dups = []
with open("../duplicate_ds.csv","r") as fp:
for line in fp.readlines()[1:]:
PRJ_ID, MST_PR, DUP_PRS, NOTE = line.split(",")
dups.append((int(PRJ_ID), int(MST_PR),
[int(item) for item in DUP_PRS.split(";")], NOTE.strip()))
pairs = []
for item in dups:
PRJ_ID, MST_PR, DUP_PRS, NOTE = item
# print PRJ_ID, MST_PR, DUP_PRS, NOTE
_pairs = omit_problem_pair(DUP_PRS, NOTE, MST_PR, PRJ_ID)
pairs.extend(_pairs)
# dups = []
# with open("../duplicate_ds.csv","r") as fp:
# for line in fp.readlines()[1:]:
# PRJ_ID, MST_PR, DUP_PRS, NOTE = line.split(",")
# dups.append((int(PRJ_ID), int(MST_PR),
# [int(item) for item in DUP_PRS.split(";")], NOTE.strip()))
# pairs = []
# for item in dups:
# PRJ_ID, MST_PR, DUP_PRS, NOTE = item
# # print PRJ_ID, MST_PR, DUP_PRS, NOTE
# _pairs = omit_problem_pair(DUP_PRS, NOTE, MST_PR, PRJ_ID)
# pairs.extend(_pairs)
cursor.execute("select prj_id, dup_pr,mst_pr from duplicate")
results = cursor.fetchall()
pairs = []
for item in results:
pairs.append(item)
return pairs
def tem_rela():

28
experiment_code/major_revision/rq2/difference/hypothesis_test.py
View File

@ -119,8 +119,12 @@ def get_raw_results():
fields = [
# "project"
'proj_age', 'prmodel_age', 'open_tasks', 'team_size',
'watchers', 'forks', 'contributors','pullreqs',
'proj_age',
# 'prmodel_age',
'open_tasks', 'team_size',
'watchers',
'forks',
# 'contributors','pullreqs',
'hotness',
# submitter
'first_pr', 'prev_pullreqs', 'prev_prs_acc',
@ -214,12 +218,12 @@ if __name__ == "__main__":
cates = {0:("Maturity",2), 2:("Workload",2), 4:("Popularity",4),8:("Hotness",1),
9:("Experience",6),15:("Standing",2), 17:("Connection",1),
18:("Size",2),20:('Length',1), 21:("Reference",1), 22:("Type",2)}
cates = {0:("Maturity",1), 1:("Workload",2), 3:("Popularity",2),5:("Hotness",1),
6:("Experience",6),12:("Standing",2), 14:("Connection",1),
15:("Size",2),17:('Length',1), 18:("Reference",1), 19:("Type",2)}
Group = {0:"Project-level characteristics",
9:"Submitter-level characteristics",
18:"Patch-level characteristics"}
6:"Submitter-level characteristics",
15:"Patch-level characteristics"}
# format ouput
for pos in range(0,len(pvalues)):
@ -231,9 +235,9 @@ if __name__ == "__main__":
effect_sizes= ["{:.3f}".format(item) for item in effect_sizes]
for pos in range(0,len(fields)):
if pos in Group:
print "\n\n\\midrule\\multicolumn{4}{@{}l}{\\textbf{%s}}\\\\"%Group[pos]
if pos in cates:
print "\n\\cdashline{1-4}[0.8pt/2pt]\\multirow{%s}{*}{%s}"%(cates[pos][1],cates[pos][0])
print "\n\n\\midrule\\multicolumn{3}{@{}l}{\\textbf{%s}}\\\\"%Group[pos]
# if pos in cates:
# print "\n\\cdashline{1-4}[0.8pt/2pt]\\multirow{%s}{*}{%s}"%(cates[pos][1],cates[pos][0])
filed, pv = fields[pos],pvalues[pos]
sig = pv_sig(pv)
@ -241,9 +245,9 @@ if __name__ == "__main__":
filed = filed.replace("_","\\_")
if filed == "prev\\_pullreqs\\_proj":
print "\\multicolumn{2}{@{}r}{\\texttt{%s}} &%s %s& %s\\\\"%(filed,pv, sig, effect_sizes[pos])
print "\\multicolumn{1}{@{}r}{\\texttt{%s}} &%s %s& %s\\\\"%(filed,pv, sig, effect_sizes[pos])
else:
print "& \\texttt{%s}&%s %s& %s \\\\"%(filed, pv , sig,effect_sizes[pos])
print "\\texttt{%s}&%s %s& %s \\\\"%(filed, pv , sig,effect_sizes[pos])
# print Cohends

224
experiment_code/major_revision/rq2/durs.csv
View File

@ -14640,7 +14640,6 @@ non-dup,241.0
non-dup,98.0
non-dup,650.0
non-dup,509.0
non-dup,1119336.0
non-dup,87421.0
non-dup,7841.0
non-dup,104.0
@ -29247,7 +29246,6 @@ non-dup,2591415.0
non-dup,3.0
non-dup,821.0
non-dup,344.0
non-dup,10978885.0
non-dup,313.0
non-dup,21243.0
non-dup,25489253.0
@ -44684,7 +44682,6 @@ non-dup,942808.0
non-dup,105.0
non-dup,357831.0
non-dup,261053.0
non-dup,792290.0
non-dup,214.0
non-dup,44.0
non-dup,176.0
@ -103814,7 +103811,6 @@ non-dup,518.0
non-dup,426.0
non-dup,101024.0
non-dup,107.0
non-dup,4331057.0
non-dup,18228.0
non-dup,493.0
non-dup,935.0
@ -108043,7 +108039,6 @@ non-dup,27842.0
non-dup,123.0
non-dup,67.0
non-dup,216.0
non-dup,151328.0
non-dup,821.0
non-dup,210.0
non-dup,56.0
@ -110224,7 +110219,6 @@ non-dup,12578053.0
non-dup,12588293.0
non-dup,267.0
non-dup,308.0
non-dup,4206957.0
non-dup,16.0
non-dup,433.0
non-dup,666.0
@ -120793,7 +120787,6 @@ non-dup,109.0
non-dup,18407761.0
non-dup,329.0
non-dup,1424.0
non-dup,5773769.0
non-dup,7692.0
non-dup,125845.0
non-dup,68.0
@ -137284,7 +137277,6 @@ non-dup,64149.0
non-dup,839.0
non-dup,548187.0
non-dup,336646.0
non-dup,208630.0
non-dup,903.0
non-dup,3844.0
non-dup,68.0
@ -146334,7 +146326,6 @@ non-dup,625.0
non-dup,1925340.0
non-dup,88.0
non-dup,84.0
non-dup,28698592.0
non-dup,5.0
non-dup,15.0
non-dup,485.0
@ -166110,7 +166101,6 @@ non-dup,313.0
non-dup,297.0
non-dup,201.0
non-dup,33043.0
non-dup,65169.0
non-dup,47.0
non-dup,172065.0
non-dup,228.0
@ -178633,7 +178623,6 @@ non-dup,2486259.0
non-dup,232.0
non-dup,1617990.0
non-dup,9987.0
non-dup,10679594.0
non-dup,1350545.0
non-dup,1556433.0
non-dup,2301.0
@ -181814,7 +181803,6 @@ non-dup,1238.0
non-dup,79263.0
non-dup,34956282.0
non-dup,15838267.0
non-dup,157894.0
non-dup,2792.0
non-dup,694932.0
non-dup,12405.0
@ -183621,7 +183609,6 @@ non-dup,50.0
non-dup,9608705.0
non-dup,354.0
non-dup,6438420.0
non-dup,6438328.0
non-dup,7514.0
non-dup,2682177.0
non-dup,883.0
@ -183716,7 +183703,6 @@ non-dup,256.0
non-dup,167556.0
non-dup,434.0
non-dup,122.0
non-dup,3154327.0
non-dup,8899.0
non-dup,436.0
non-dup,237.0
@ -184405,7 +184391,6 @@ non-dup,90.0
non-dup,741235.0
non-dup,23.0
non-dup,377.0
non-dup,13848921.0
non-dup,363.0
non-dup,2127748.0
non-dup,263.0
@ -184443,7 +184428,6 @@ non-dup,454.0
non-dup,108.0
non-dup,613.0
non-dup,401.0
non-dup,31281318.0
non-dup,280.0
non-dup,456095.0
non-dup,2413533.0
@ -185754,7 +185738,6 @@ non-dup,21076.0
non-dup,294.0
non-dup,138.0
non-dup,486750.0
non-dup,509396.0
non-dup,163.0
non-dup,186.0
non-dup,208.0
@ -185842,7 +185825,6 @@ non-dup,2053459.0
non-dup,258593.0
non-dup,646.0
non-dup,174.0
non-dup,5346614.0
non-dup,417.0
non-dup,2383.0
non-dup,71.0
@ -187064,7 +187046,6 @@ non-dup,4123274.0
non-dup,166.0
non-dup,138.0
non-dup,8304844.0
non-dup,12582825.0
non-dup,15558057.0
non-dup,586.0
non-dup,19938.0
@ -187971,7 +187952,6 @@ non-dup,712.0
non-dup,45.0
non-dup,65.0
non-dup,723.0
non-dup,35683.0
non-dup,36089.0
non-dup,6249.0
non-dup,41.0
@ -191994,7 +191974,6 @@ non-dup,1035.0
non-dup,505.0
non-dup,146.0
non-dup,92.0
non-dup,4614512.0
non-dup,2545.0
non-dup,623.0
non-dup,245.0
@ -207658,7 +207637,6 @@ non-dup,79087.0
non-dup,1219.0
non-dup,727.0
non-dup,407307.0
non-dup,251963.0
non-dup,263547.0
non-dup,255732.0
non-dup,26850.0
@ -208009,7 +207987,6 @@ non-dup,904.0
non-dup,24572.0
non-dup,20862.0
non-dup,4010.0
non-dup,137625.0
non-dup,265707.0
non-dup,309.0
non-dup,499329.0
@ -214156,7 +214133,6 @@ non-dup,2464.0
non-dup,50.0
non-dup,6661.0
non-dup,125564.0
non-dup,429516.0
non-dup,54.0
non-dup,51.0
non-dup,72.0
@ -214750,7 +214726,6 @@ non-dup,381.0
non-dup,4611.0
non-dup,5724.0
non-dup,159.0
non-dup,8749589.0
non-dup,3905900.0
non-dup,125068.0
non-dup,92.0
@ -263557,7 +263532,6 @@ non-dup,9.0
non-dup,4.0
non-dup,5.0
non-dup,5.0
non-dup,143184.0
non-dup,5.0
non-dup,122.0
non-dup,7.0
@ -271430,7 +271404,6 @@ non-dup,1808.0
non-dup,507.0
non-dup,721.0
non-dup,45652.0
non-dup,318588.0
non-dup,129.0
non-dup,1626.0
non-dup,7.0
@ -292725,7 +292698,6 @@ non-dup,149.0
non-dup,195.0
non-dup,3136.0
non-dup,47.0
non-dup,180722.0
non-dup,1192.0
non-dup,195.0
non-dup,12.0
@ -308170,7 +308142,6 @@ non-dup,1339.0
non-dup,38.0
non-dup,198.0
non-dup,1763.0
non-dup,4266560.0
non-dup,249.0
non-dup,13.0
non-dup,1981.0
@ -313326,7 +313297,6 @@ non-dup,197.0
non-dup,5365.0
non-dup,446.0
non-dup,1386.0
non-dup,5472166.0
non-dup,1485.0
non-dup,597.0
non-dup,103.0
@ -313789,101 +313759,133 @@ non-dup,666.0
non-dup,328.0
non-dup,460.0
non-dup,145.0
inc_dup,799349.0
inc_dup,9839.0
inc_dup,3204.0
inc_dup,27275.0
inc_dup,264438.0
inc_dup,102285.0
inc_dup,2243.0
inc_dup,792290.0
inc_dup,1460023.0
inc_dup,180.0
inc_dup,4266560.0
inc_dup,180722.0
inc_dup,442499.0
inc_dup,5472166.0
inc_dup,501643.0
inc_dup,1626752.0
inc_dup,10284.0
inc_dup,1370.0
inc_dup,66476.0
inc_dup,4331057.0
inc_dup,2959.0
inc_dup,125568.0
inc_dup,1109336.0
inc_dup,79541.0
inc_dup,7851347.0
inc_dup,923039.0
inc_dup,4088364.0
inc_dup,1962463.0
inc_dup,480667.0
inc_dup,153585.0
inc_dup,8386055.0
inc_dup,89709.0
inc_dup,619156.0
inc_dup,939791.0
inc_dup,16621213.0
inc_dup,6209.0
inc_dup,494.0
inc_dup,1096.0
inc_dup,102285.0
inc_dup,137625.0
inc_dup,251963.0
inc_dup,2243.0
inc_dup,5773769.0
inc_dup,952233.0
inc_dup,2511212.0
inc_dup,20870.0
inc_dup,526260.0
inc_dup,1386.0
inc_dup,20870.0
inc_dup,2511212.0
inc_dup,952233.0
inc_dup,21028327.0
inc_dup,15824468.0
inc_dup,12521502.0
inc_dup,159764.0
inc_dup,6041191.0
inc_dup,16256058.0
inc_dup,66684.0
inc_dup,27105.0
inc_dup,6587.0
inc_dup,5274414.0
inc_dup,23864141.0
inc_dup,1986512.0
inc_dup,2404346.0
inc_dup,36287616.0
inc_dup,51268817.0
inc_dup,339060.0
inc_dup,29967323.0
inc_dup,96908.0
inc_dup,234772.0
inc_dup,347765.0
inc_dup,60974.0
inc_dup,7252764.0
inc_dup,10397.0
inc_dup,54932.0
inc_dup,10765.0
inc_dup,480667.0
inc_dup,1962463.0
inc_dup,13848921.0
inc_dup,31281318.0
inc_dup,4088364.0
inc_dup,3551288.0
inc_dup,12582825.0
inc_dup,7851347.0
inc_dup,8386055.0
inc_dup,619156.0
inc_dup,6438328.0
inc_dup,509396.0
inc_dup,89709.0
inc_dup,6438328.0
inc_dup,35683.0
inc_dup,153585.0
inc_dup,939791.0
inc_dup,923039.0
inc_dup,41383.0
inc_dup,2480.0
inc_dup,15220.0
inc_dup,168195.0
inc_dup,318.0
inc_dup,143176.0
inc_dup,9050618.0
inc_dup,2984177.0
inc_dup,3912641.0
inc_dup,630768.0
inc_dup,7520791.0
inc_dup,442499.0
inc_dup,180.0
inc_dup,1626752.0
inc_dup,501643.0
inc_dup,1460023.0
inc_dup,10397.0
inc_dup,7252764.0
inc_dup,60974.0
inc_dup,10765.0
inc_dup,54932.0
inc_dup,2959.0
inc_dup,10284.0
inc_dup,1370.0
inc_dup,66476.0
inc_dup,42744.0
inc_dup,19653106.0
inc_dup,41383.0
inc_dup,2480.0
inc_dup,168195.0
inc_dup,15220.0
inc_dup,143176.0
inc_dup,318.0
inc_dup,3551288.0
inc_dup,13088.0
inc_dup,799349.0
inc_dup,3204.0
inc_dup,9839.0
inc_dup,10978885.0
inc_dup,4265766.0
inc_dup,29967323.0
inc_dup,339060.0
inc_dup,208630.0
inc_dup,264438.0
inc_dup,143184.0
inc_dup,27275.0
inc_dup,66684.0
inc_dup,318588.0
inc_dup,28698592.0
inc_dup,65169.0
inc_dup,51268817.0
inc_dup,96908.0
inc_dup,234772.0
inc_dup,347765.0
inc_dup,1119336.0
inc_dup,4206957.0
inc_dup,6041191.0
inc_dup,159764.0
inc_dup,151328.0
inc_dup,6587.0
inc_dup,1986512.0
inc_dup,1986512.0
inc_dup,27105.0
inc_dup,13088.0
inc_dup,5274414.0
inc_dup,23864141.0
inc_dup,19653106.0
inc_dup,42744.0
inc_dup,12724060.0
inc_dup,12724060.0
ovl_mst,6099318.0
inc_dup,16621213.0
inc_dup,6209.0
inc_dup,36287616.0
inc_dup,2404346.0
inc_dup,79541.0
inc_dup,1109336.0
inc_dup,16256058.0
inc_dup,4614512.0
inc_dup,12521502.0
inc_dup,15824468.0
inc_dup,21028327.0
inc_dup,494.0
inc_dup,1096.0
ovl_mst,88.0
ovl_mst,283.0
ovl_mst,237.0
ovl_mst,900991.0
ovl_mst,179.0
ovl_mst,395600.0
ovl_mst,8749589.0
ovl_mst,6099318.0
ovl_mst,429516.0
ovl_mst,736.0
ovl_mst,5346614.0
ovl_mst,10679594.0
ovl_mst,18727876.0
ovl_mst,157894.0
ovl_mst,3154327.0
ovl_mst,18727876.0
ovl_mst,841111.0
ovl_mst,736.0
ovl_mst,236.0
ovl_mst,223.0
ovl_mst,63555.0
ovl_mst,110.0
ovl_mst,5621408.0
ovl_mst,839.0
ovl_mst,110.0
ovl_mst,63555.0
ovl_mst,179.0
ovl_mst,395600.0
ovl_mst,236.0
ovl_mst,777.0
ovl_mst,88.0
ovl_mst,237.0
ovl_mst,283.0

Can't render this file because it is too large.

4
experiment_code/major_revision/rq2/formate_ttest_result.py
View File

@ -19,8 +19,8 @@ with open("ttest_result.txt","r") as fp:
# 2 mst_pr
# 3 non-dup
group = {
"3" : "NON",
"2" : "OVL",
"3" : "OVL",
"2" : "NON",
"1" : "INC"
}

7
experiment_code/major_revision/rq3/2_pr_metrics.py
View File

@ -48,10 +48,11 @@ def test_inclusion(prs):
for pr in prs:
d_id, repo_id, pr_num, dt = pr
ft = _tested(repo_id, pr_num)
if ft == 1:
if ft:
test_inclusion = 1
else:
test_inclusion = 0
print ft, test_inclusion
cursor.execute("update rq3_metrics set test_inclusion=%s where id=%s",
(test_inclusion, d_id))
conn.commit()
@ -127,8 +128,8 @@ if __name__ == "__main__":
prs = cursor.fetchall()
# change_size(prs)
# test_inclusion(prs)
text_len(prs)
test_inclusion(prs)
# text_len(prs)
# num_commits(prs)
# issue_tag(prs)

4
experiment_code/major_revision/rq3/3_review_metrics.py
View File

@ -533,12 +533,12 @@ if __name__ == "__main__":
cursor.execute("select id, repo_id, pr_num, determined_at from rq3_metrics order by repo_id,pr_num")
prs = cursor.fetchall()
# comments(prs)
comments(prs)
# sentiment(prs)
# status(prs)
print None_check
patch_revision(prs)
# patch_revision(prs)
# senti(prs)

BIN
experiment_code/major_revision/rq3/decision.xls
View File

Binary file not shown.

2
experiment_code/major_revision/rq3/utils.py
View File

@ -137,7 +137,7 @@ def _tested(prj_id, pr_num):
files = cursor.fetchall()
if len(files) == 0:
return None
return False
for df in files:
file_type = MyfiletypeJudger(df[0], lan,rn)

BIN
experiment_code/major_revision/初步修订计划-english.docx
View File

Binary file not shown.

19
ref.bib
View File

@ -1,4 +1,15 @@
@Misc{intor-avoid,
howpublished = {\url{https://scikit-learn.org/dev/developers/contributing.html}},
note = {Accessed: 2020-6-11},
title = {Contributing to Scikit-learn},
}
@Misc{cntor-avoid,
howpublished = {\url{https://github.com/pandas-dev/pandas/pull/12576}},
note = {Accessed: 2020-6-11},
title = {A pull request from the project Pandas},
}
@inproceedings{Adams2014Do,
title={Do developers feel emotions? an exploratory analysis of emotions in software artifacts},
@ -8,6 +19,12 @@
pages={262--271},
}
@inproceedings{Jensen2007Role,
title={Role Migration and Advancement Processes in OSSD Projects: A Comparative Case Study},
author={Jensen, Chris and Scacchi, Walt},
booktitle={Software Engineering, 2007. ICSE 2007. 29th International Conference on},
year={2007},
}
@inproceedings{zhou2018identifying,
title={Identifying features in forks},
@ -964,8 +981,6 @@ pages={459--462},
booktitle={Proceedings of 2017 IEEE/ACM 14th International Conference on Mining Software Repositories},
year={2017}
}
@inproceedings{beller2016analyzing,
title={Analyzing the state of static analysis: A large-scale evaluation in open source software},

BIN
resources/dur_cmp.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 47 KiB