Keynote Speaker

James D. Herbsleb School of Computer Science Carnegie Mellon University

Sponsors

Deadlines

Organizing Committee

Advisory Committee

Program Committee

Preparing for the Workshop

Keynote Speaker

James Herbsleb is a Professor in the School of Computer Science at Carnegie Mellon University. His research interests focus on collaboration and coordination in software and systems engineering projects. His research iterates over empirical studies, theory development, and design and deployment of technology. Before accepting a position at CMU, Herbsleb led the Bell Labs Collaboratory project, focused on understanding and solving issues in geographically-distributed software development. He holds a PhD in psychology and a JD in law from the University of Nebraska, as well as a MS in computer science from the University of Michigan, where he also completed a post-doctoral fellowship. His early ambition was to be a 21st-century Renaissance man, until he encountered normal forms in database design, at which point he realized there were some things he didn't want to know.

• Experiences from Replicating a Case Study to Investigate Reproducibility of Software Development

  Audris Mockus, Bente Anda, and Dag I.K. Sjøberg  Show Abstract  PDF

  A fundamental question for software engineering in general, and for empirical software engineering in particular, is the reproducibility of software development: which aspects of a software development process and product are reproducible and which parts depend on the creativity of developers? We have investigated the reproducibility of software development projects and products in the context of four commercial companies developing independently an identical medium-sized document management system. We carried out four instances (replications) of a case study in which team size and experience, functional requirements, and customer interactions were controlled for, while the cost and duration of the project varied. This allowed us to gain experience with replicating software engineering case studies. The practical experiences are in particular related to 1) choice of replication cases according to type of replication, and 2) data collection in replicated case studies. Furthermore, the specific results from the study on reproducibility, that show low reproducibility for important aspects of software development even within a relatively restricted context, support the need for more case studies in software engineering, and shows that a large number of replications may be needed to obtain clear results.

• Beyond Replication: An example of the potential benefits of replicability in the Mining Software Repositories Community

  Gregorio Robles and Daniel M. German  Show Abstract  PDF

  While in theory the mining software repositories is an area where replication is easier to perform than for other empirical software engineering fields, a review of papers presented at the MSR workshop/working conference shows that the research studies presented do not satisfy the requirements for an easy replication. In this paper, we present some possibilities that replicability may provide to this community that go beyond the verification of results presented in the original study.

• A Case for Replication: Synthesizing Research Methodologies in Software Engineering

  Jonathan L. Krein and Charles D. Knutson  Show Abstract  PDF

  Software Engineering (SE) problems are—from both practical and theoretical standpoints—immensely complex, involving interactions between technical, behavioral, and social forces. In an effort to dissect this complexity, SE researchers have incorporated a variety of research methods. Recently, the field has entered a paradigm shift—a broad awakening to the social aspects of software development. As a result, and in concert with an ongoing struggle to establish SE research as an empirical discipline, SE researchers are increasingly appropriating methodologies from other fields. In the wake of this self-discovery, the field is entering a period of methodological flux, during which it must establish for itself effective research practices. We present a unifying framework for organizing research methods in SE. In the process of elucidating this framework, we dissect the current literature on replication methods and place replication appropriately within the framework. We also further clarify, from a high level and with respect to SE, the mechanisms through which science builds usable knowledge.

• Replication, Reproduction and Re-analysis: Three ways for verifying experimental findings

  Omar S. Gómez G., Natalia Juristo, and Sira Vegas  Show Abstract  PDF

  The verification of a previously observed finding is important. Being able to verify a finding provides some guarantee that it is not product of chance but that the observed phenomenon is stable or regular. The most well done way of verifying experimental findings is through replication. But, is this the only method for verifying the observations of an experiment? We have studied different ways of verifying experimental findings that are used in other disciplines. From the 18 replication classifications that we located, we have identified three methods for verifying a finding. Each of these methods fulfil a particular verification purpose.

• Rethinking Replication in Software Engineering: Can We See the Forest for the Trees?

  Mika V. Mäntylä, Casper Lassenius, and Jari Vanhanen  Show Abstract  PDF

  In this paper, we argue that the concept of replication of empirical studies in software engineering should be understood more broadly than it currently is. In particular, the replication of case studies and surveys as a way of validating and extending theories should be incorporated in the mainstream view of replication, which at present is mostly focused on controlled experiments. A small-sample study of papers in IEEE Transactions on Software Engineering shows that about 10% of studies published in 2009 can be considered replications. However, none of these was self-labeled as replication. We think that the authors believed that labeling their work as replication might decrease its value in the eyes of reviewers and editors. We conclude that there is no acute shortage of replication studies in software engineering if taking a broader viewpoint to replication, but the definition and valuation of 'replication studies' need to be re-evaluated in the software engineering community.

• Towards Reporting Guidelines for Experimental Replications: A Proposal

  Jeffrey C. Carver  Show Abstract  PDF

  The value of experimental replications has been well established. In order for the replicating researcher and the community to receive the greatest benefit from a replication, the right information about it must be published. This paper proposes publishing guidelines to increase the value of experimental replications. First, a review of some published replications highlights the variation in current publishing practice. Then, a set of guidelines are proposed. The goal of this paper is to provide a starting point for a discussion that will formalize and publish a set of guidelines.

• Hidden Evidence Behind Useless Replications

  Oscar Dieste, Enrique Fernandez, Ramón Garcia, and Natalia Juristo  Show Abstract  PDF

  Experiments that are run with few experimental subjects are often considered not to be very reliable and deemed, as a result, to be useless with a view to generating new knowledge. This belief is not, however, entirely correct. Today we have tools, such as meta-analysis, that we can use to aggregate small-scale experiments and output results that are equivalent to experiments run on large samples that are therefore reliable. The application of meta-analysis can overcome some of the obstacles that we come up against when running software engineering experiments (such as, for example, the practitioner availability problem).

• Transitioning from lab studies to large-scale studies: Emerging results from a literal replication

  Remo Ferrari, Oliver Sudmann, Christian Henke, Jens Geisler, Wilhelm Schafer, and Nazim H. Madhavji  Show Abstract  PDF

  Replication of studies in Software Engineering is considered important, but is largely neglected. Because of the lack of published replicated literature, there are few established guidelines for researchers wanting to conduct replicated studies. Specifically, guidelines for transitioning from laboratory studies to large-scale studies are nonexistent. Previously, we replicated a laboratory study in the banking domain by conducting an extended, large-scale case study on an innovative rail project in Germany, investigating the role of an existing systems architecture on requirements decisions. In this short paper, we present our preliminary analysis of our transitioning experiences from conducting these two studies. From our experiences, we derive a set of lessons learnt and recommendations that can be used by other researchers wanting to transition from lab studies to studies in industrial settings.

• Threats to Validity in Analysis of Language Fragmentation on SourceForge Data

  Alexander C. MacLean, Landon J. Pratt, Jonathan L. Krein, and Charles D. Knutson  Show Abstract  PDF

  Reaching general conclusions through analysis of SourceForge data is difficult and error prone. Several factors conspire to produce data that is sparse, biased, masked, and ambiguous. We explore these factors and the negative effect that they had on the results of "Impact of Programming Language Fragmentation on Developer Productivity: a SourceForge Empirical Study." In addition, we question the validity of evolutionary or temporal analysis of development practices based on this data.

• How Developers Participate in Open Source Projects: a Replicate Case Study on JBossAS, JOnAS and Apache Geronimo

  Xiujuan Ma, Minghui Zhou, and Hong Mei  Show Abstract  PDF

  Ten years passed from the study of how open source projects develop by Mockus et al.. More and more commercial companies have been joining and supporting OSS projects. Have they changed the essence of OSS development? Do these hypotheses still hold? In this paper we replicate Apache study to understand the current situation of open source development, in particular, the developers' participation in the commercially supported OSS projects. We investigate three OSS projects from J2EE platform domain:JBossAS, JOnAS, and Apache Geronimo, which all have a commercial background. By answering five research questions in Apache study, we evaluate four hypotheses finding the following: the number of the core developers was around 15 in JOnAS and Geronimo. But JBossAS had a larger core team; all the three projects haven't achieved a large number of outside contributors to free the core developers from finding and repairing defects; most of the none-core developers participated in the project with contributing new functionality; they reacted faster to customer problems than the commercial projects but slower than Apache. It appears that the commercial support can attract more volunteers to the projects, and drives the developers be more sensitive to the "priority" filed provided by the issue reporters.