On the Fulfillment of Coordination Requirements in Open-Source Software Projects:
An Exploratory Study

Coordination among developers is crucial in large-scale open-source software projects, where developers are often distributed across the entire planet. By assessing the alignment of collaboration and communication in such software projects in terms of coordination requirements, we can estimate whether a state of socio-technical congruence is achieved, which is associated with software quality and project success. By means of an empirical study on a substantial set of large-scale open-source software projects (including OpenSSL, Git, and LLVM)—altogether making up over 180 years of development history—we aim at shedding light at this issue. Compared to the state of the art in this research area, we do not only identify coordination requirements arising from files and functions only, but also those arising from features. This way, we take a more semantic view on this phenomenon. We found that open-source developers fulfill coordination requirements on purpose, but mostly those coordination requirements arising from coupled source-code artifacts, while they resolve simpler ones independently. Furthermore, we found that neither of the considered abstraction levels of source-code artifacts (files, functions, features) is more suitable as constructional argument for coordination requirements with respect to their fulfillment. This finding strongly indicates that features do not play an as important role in the development process as expected and commonly believed by the research community in the area of feature-oriented and feature-driven development. Finally, we identified notable evolutionary trends in the fulfillment of coordination requirements and showed that far-reaching social events have a huge impact on their fulfillment, both negatively and positively. The key findings of our empirical study are that socio-technical relations are important to understand open-source development communities and that the incorporation of different abstraction levels for developer collaboration does yield important insights to further improve the evolution in open-source software projects.

Keywords: coordination requirements socio-technical congruence social-network analysis coronet Codeface open-source software systems configurable systems software product lines feature-oriented software development

Research Questions

RQ₁

Does developer communication align with artifact-based coordination requirements in real-world OSS projects such that the coordination requirements are fulfilled?

RQ₂

Does developer communication align better with feature-based coordination requirements than with function-based or file-based coordination requirements?

RQ₃

Does the degree of fulfillment of coordination requirements change for different artifact types during project evolution??

Hypotheses

RH₁

A high number of developer pairs collaborating on the same artifacts do exchange e-mails on the same threads of the mailing list, such that coordination requirements arising from any type of artifact are fulfilled not only by chance.

RH₂

The fraction of fulfilled coordination requirements is lower for the square motif than for the triangle motif, independent of the observed artifact abstraction.

RH₃

The fraction of fulfilled coordination requirements differs for the different artifacts, and is significantly highest for abstraction level of features.

RH₄

In later stages of development, the fraction of fulfilled coordination requirements is higher than in earlier stages for all motifs and artifacts.

Coordination-Requirement Networks and Motifs

To analyze the fulfillment of coordination requirements in a software project, we construct coordination-requirement networks, which we can analyze with network-analytic methods. We show an exemplary coordination-requirement network in Figure 1. Formally, such a network is defined as an undirected graph G = (D ∪ A, E), where we encode developers (, set D) and artifacts (, set A) as vertices; E is the set of edges among the vertices. We encode the following three relations in the edges:

• Developer–artifact relation (): Developers work on code artifacts while committing to the project’s version control system. Such artifacts may be files, functions, or features (which may crosscut the file and function decomposition).
• Artifact–artifact relation (): Artifacts can be related in various ways, giving raise to interdependencies. We consider co-changes to describe logical coupling among artifacts. The term “co-changes” refers to artifacts that are concurrently changed in a single commit in the project’s version control system.
• Developer–developer relation (): We consider contributions of the developers to their project’s mailing list: In line with the seminal work by Bird et al., we assume that two developers coordinate their work iff they contribute to the same thread on the mailing list [8].

To automatically identify coordination requirements in coordination-requirement networks, we encode coordination requirements or, rather, the patterns they represent as network motifs. Network motifs are recurrent sub-graphs in a given network [75]. Motifs can be described formally as a set of vertices (e.g., {d₁, d₂, a₁}) with specific edges connecting them. We show two network motifs for coordination requirements in Figure 2, the triangle motif and the square motif.

Abstraction Levels for Source-Code Artifacts

In previous work [14, 13], researchers have tracked concurrent contributions of developers on the same file to derive coordination requirements. We conjecture that this view may be too technical to capture the richness of coordination. Thus, we introduce the two code-artifact abstractions function and feature to infer coordination requirements at different levels of abstraction. Although these abstraction levels are based on heuristics, they have been shown to be reliable in multiple previous studies [51, 55, 43, 44]. Additionally, with regard to network constructions, the abstraction file has been shown to produce dense networks that are known to hinder community detection [10, 43] and, thus, represent more precise coordination relations among developers. It has been already shown that a function-level view is more accurate [44].

For illustration, we show how strongly the choice of abstraction level influences the extraction of coordination requirements by means of the triangle motif and its manifestation in the source-code excerpt listed in Figure 3 (show/hide). We show the resulting coordination-requirement networks in Figure 4.

Show source-code example (Figure 3) →

diff --git a/actions.c b/actions.c
index d4ea8ff..ecb9f59 100644
--- a/actions.c
+++ b/actions.c
@@ -1,8 +1,11 @@ Changes by Dev C
void delete (struct DBconnection *DBconn,
          char *command) {
 //...
+ #ifdef PERSIST
+    persist();
+ #endif
}

 #ifdef FEATURE_LOCKING
void lockOnAction(struct DBaction *action) {

@@ -11,12 +11,12 @@ Changes by Dev C
-    // old code
+    // Dev C makes changes here

@@ -13,21 +13,21 @@ Changes by Dev B
-    if (data == NULL) {
+    if (data != NULL) {
      // ...
 }

 // more code

}
 #endif

diff --git a/db.c b/db.c
index 130f79c..d1c26b1 100644
--- a/db.c
+++ b/db.c
@@ -1,6 +1,6 @@ Changes by Dev A
 #ifdef PERSIST
-void persist() {
-    // old code
+void persist(char *filename) {
+    // completely rewritten code
}
 #endif

@@ -7,10 +7,10 @@ Changes by Dev B
void execute(struct DBConn *conn,
          struct DBAction *action) {
-    // old code
+    // code with bugfixes
}

Statistics and Formulas

To analyze the alignment of the email-based developer coordination and the actual artifact-based collaboration, we measure the fraction of fulfilled coordination requirements. Given a coordination-requirement network constructed using one type of code artifact (i.e., file, function, or feature) and a motif m to identify coordination requirements, we define the fraction frac_cr(a, m) of fulfilled coordination requirements as follows:

frac_cr(a, m) = |cr_full(a, m)| / |cr_found(a, m)|, where

cr_found(a, m) = { c | matched instance c of motif m for artifact a in the current network } and
cr_full(a, m) = { c_f | c_f ∈ cr_found(a, m), c_f is fulfilled },

For data extraction, we mainly use the tool Codeface. Based on the Codeface results, we construct and analyze coordination-requirement networks using our network-construction library coronet and a set of self-written R scripts. Our script setup is available in the Downloads section.

Overview on Subject Systems

Hypothesis RH₁

Sensitivity Analysis following Kossinets (2006)

We performed a sensitivity analysis following Kossinets (2006) to investigate on the stability of our results. In detail, we used the simulation algorithm "BSPC" (boundary specification problem for contexts) to simulate the absence of coordination effort from the mailing list (which may occur on different platforms such as face-to-face meetings or chats instead) and, thus, incomplete information sources (i.e., mailing-list data) – similar to the null models (see Section 3.2.4). The algorithm removes a defined number of random e-mail threads before constructing analyzable coordination-requirement networks and calculates the metrics as previously defined. To this end, for the projects BusyBox, Git, LLVM, and OpenSSL, we randomly removed 10, 20, …, 90 percent of the e-mail threads, performed 25 iterations for better randomization, calculated mean values, and analyzed the final results. In short, we found for the selected projects that the removal of 10% of all e-mail threads produces a relative error of about 15% for frac_cr, across all revision ranges and for all motifs and source-code artifacts. With 20% of all e-mail threads being randomly removed, the metric exhibits a relative error of about 25%, on average. Results can be opened/displayed in the table below. These results indicate that the absence of crucial developers may have an immediate and extensive effect on most projects and emphasize that any coordination effort is important to fulfill coordination requirements.

Hypothesis RH₂

Hypothesis RH₃

Hypothesis RH₄

Apache HTTP

BusyBox

FFmpeg

Git

LLVM

OpenSSL

PostgreSQL

QEMU

U-Boot

Wine

Apache HTTP

BusyBox

FFmpeg

Git

LLVM

OpenSSL

PostgreSQL

QEMU

U-Boot

Wine

Downloadable assets:

• Subject-system details: subject-systems-details.ods
• Codeface configuration files: network-coordination-requirements_configurations.zip
• Analysis scripts: network-coordination-requirements_scripts.zip
• Statistical results: network-coordination-requirements_stats.zip
• History plots (Hypothesis RH₄): network-coordination-requirements_plots.zip
• Plots for sensitivity analysis (Hypothesis RH₁): network-coordination-requirements_plots-sensitivity.zip

To reproduce the data for an individual subject project, the data needs to be processed by Codeface and codeface-extraction first – please use the configuration files provided above. Afterwards, the output data can be processed using our analysis scripts. The main script of our analysis is analysis.R and needs to be used in each and every stage of the analysis (show/hide command-line interface).

usage: analysis.R [-h] [-s SELECTION_PROCESS] [-w SPLIT_WINDOW]
                  [--sliding-window] [-d CODEFACE_DATA]
                  [--null-model NULL_MODEL] [--no-remove-core]
                  [--complete-rerun] [--bootstrap-packrat]
                  [--loglevel LOGLEVEL]
                  script casestudy artifact artifact_relation

positional arguments:
  script                The subscript to execute
  casestudy             Casestudy name as in Codeface-data folder name
  artifact              Artifact type to use in the analysis
  artifact_relation     Artifact-relation type to use in the analysis

optional arguments:
  -h, --help            show this help message and exit
  -s SELECTION_PROCESS, --selection-process SELECTION_PROCESS
                        The selection process for revision windows [default
                        "releases"]
  -w SPLIT_WINDOW, --split-window SPLIT_WINDOW
                        The time-window length used for data splitting (e.g.,
                        "3 months" and "2 weeks") [default "3 months"]
  --sliding-window      Do you want overlapping time windows (i.e., sliding-
                        window approach)? [default "False"]
  -d CODEFACE_DATA, --codeface-data CODEFACE_DATA
                        Path to Codeface data [default "/local/hunsen/projects
                        /codeface-data/"]
  --null-model NULL_MODEL
                        The specific null model to use
                        ('rewire'|'random'|'errg') [default "rewire"]
  --no-remove-core      Remove core authors from null model? [default "False"]
  --complete-rerun      Do a complete re-run of the analysis? [default
                        "False"]
  --bootstrap-packrat   Bootstrap packrat library?
  --loglevel LOGLEVEL   Log level

Overall, there are five different stages in our analysis, which need to be run consecutively and which (mostly) run for one subject system at a time (for each, data output is cached appropriately):

1. data: run the empirical analysis by constructing coordination-requirement networks and searching for motifs
2. null: construct null-model networks and search for motifs
3. sensitivity: construct sensitivity model following Kossinets (2006) and search for motifs
4. stats: run all statistical tests and construct corresponding plots (independent of configured subject system)
5. history-plots: construct the history plots for Hypothesis RH₄

Based on the configuration files provided by us, the following parameters should be given to each analysis stage (see command-line interface above):

• script: a stage as described above
• casestudy: subject-system name as in Codeface configuration
• artifact: the artifact abstraction level to use
• artifact_relation: must to be set to "cochange" to reproduce the study
• -d CODEFACE_DATA: set to path where Codeface output is placed
• -w "3 months": use three-month revision ranges
• -s "threemonth" (this is an abstraction level in Codeface to structure analyses)
• --null-model="rewire": use the null model based on rewiring
• -no-remove-core: Do not remove core developers in null-model analysis

As a consequence, an exemplary call to gather feature-based empirical data for Apache HTTP looks like this:

Rscript analysis.R data apache-http feature cochange -d "/path/to/codeface-data/" -s "threemonth" -w "3 months" --loglevel "DEBUG"

If you have any questions regarding this paper or any other related project, please do not hesitate to contact us:

• Claus Hunsen (University of Passau, Passau, Germany)
• Janet Siegmund (Chemnitz University of Technology, Chemnitz, Germany)
• Sven Apel (Saarland University, Saarbrücken, Germany)