James D. Herbsleb
School of Computer Science
Carnegie Mellon University
We would like to thank InfoTrax Systems, Inc. for reducing the registration fee by 30% and for providing the workshop dinner.
The registration price reduction
is available to the first 40 registrants
. Register for this workshop through the ICSE
We would also like to thank SirsiDynix for sponsoring flash drives on which to distribute the proceedings.
Submission: 18 March 2010 (Extended)
Notification: 8 April 2010
Camera Ready: 15 April 2010
* To avoid conflicting with ESEM
deadlines, we have extended our submissions deadline two weeks. We hope this helps!
Natalia Juristo – Universidad Politécnica de Madrid, Spain
Charles Knutson – Brigham Young University, USA
Jonathan Krein – Brigham Young University, USA
Lutz Prechelt – Freie Universität Berlin, Germany
Lionel Briand – Simula Research Laboratory, Norway
Audris Mockus – Avaya Labs Research, USA
Dieter Rombach – University of Kaiserslautern & Fraunhofer IESE, Germany
Christian Bird – University of California Davis, USA
Andrew Brooks – University of Akureyri, Iceland
Jeffrey Carver – University of Alabama, USA
Marcus Ciolkowski – Fraunhofer IESE, Germany
Kevin Crowston – Syracuse University, USA
Daniel Delorey – Google, Inc., USA
Harald Gall – University of Zurich, Switzerland
Daniel German – University of Victoria, Canada
Jesus Gonzalez-Barahona – Universidad Rey Juan Carlos, Spain
Alicia Grubb – University of Toronto, Canada
Andreas Jedlitschka – Fraunhofer IESE, Germany
James Miller – University of Alberta, Canada
Dietmar Pfahl – Simula Research Laboratory & University of Oslo, Norway
Marc Roper – University of Strathclyde, UK
Carolyn Seaman – Fraunhofer CESE & University of Maryland, USA
Janice Singer – National Research Council, Canada
Megan Squire – Elon University, USA
Sira Vegas – Universidad Politécnica de Madrid, Spain
Claes Wohlin – Blekinge Institute of Technology, Sweden
Murray Wood – University of Strathclyde, UK
Note: This list is a bit out of date. Nevertheless, if you are interested in learning more about issues of replication in empirical software engineering research, many of the resources below are excellent sources. You may want to start with the dark-bulleted items. Also consider checking out past proceedings of the RESER workshop, as well as the workshop summaries published in Software Engineering Notes (SEN).
We have tried to make this a comprehensive, yet concise list, but if there's other relevant work that you feel should be included, please email us the reference and category at jonathankrein [at] byu [dot] net.
Victor R. Basili, Forrest Shull, and Filippo Lanubile. Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4):456–473, 1999.
Andrew Brooks, Marc Roper, Murray Wood, John Daly, and James Miller. Replication's role in software engineering. In Forrest Shull, Janice Singer, and Dag I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 365–379. Springer, 2008.
Saul Greenberg and Bill Buxton. Usability evaluation considered harmful (some of the time). In 26th Annual SIGCHI Conference on Human Factors in Computing Systems, pages 111–120, New York, NY, USA, 2008. ACM.
Natalia Juristo and Sira Vegas. Using differences among replications of software engineering experiments to gain knowledge. In 3rd International Symposium on Empirical Software Engineering and Measurement, pages 356–366, Washington, DC, USA, 2009. IEEE Computer Society.
Barbara A. Kitchenham. The role of replications in empirical software engineering—a word of warning. Empirical Software Engineering: An International Journal, 13(2):219–221, 2008.
Jonathan Lung, Jorge Aranda, Steve Easterbrook, and Gregory Wilson. On the difficulty of replicating human subjects studies in software engineering. In 30th International Conference on Software Engineering, pages 191–200, New York, NY, USA, 2008. ACM.
Forrest Shull, Victor Basili, Jeffrey Carver, José Maldonado, Guilherme Travassos, Manoel Mendonça, and Sandra Fabbri. Replicating software engineering experiments: Addressing the tacit knowledge problem. International Symposium on Empirical Software Engineering, 0:7, 2002.
Forrest Shull, Jeffrey Carver, Sira Vegas, and Natalia Juristo. The role of replications in empirical software engineering. Empirical Software Engineering: An International Journal, 13(2):211–218, 2008.
Marcelo Cataldo, Audris Mockus, Jeffrey A. Roberts, and James D. Herbsleb. Software dependencies, work dependencies, and their impact on failures. IEEE Transactions on Software Engineering, 35(6):864–878, 2009.
John W. Daly. Replication and a Multi-Method Approach to Empirical Software Engineering Research. PhD thesis, Department of Computer Science, University of Strathclyde, Glasgow, U.K., 1996.
Trung T. Dinh-Trong and James M. Bieman. The FreeBSD project: A replication case study of open source development. IEEE Transactions on Software Engineering, 31(6):481–494, 2005.
Adam Porter, Lawrence Votta Jr., and Victor Basili. Comparing detection methods for software requirements inspections: a replicated experiment. IEEE Transactions on Software Engineering, 21(6):563–575, 1995.
Murray Wood, Marc Roper, Andrew Brooks, and James Miller. Comparing and combining software defect detection techniques: a replicated empirical study. SIGSOFT Software Engineering Notes, 22(6):262–277, 1997.
Magne Jørgensen and Dag Sjøberg. Generalization and theory-building in software engineering research. In 8th International Conference on Empirical Assessment in Software Engineering, pages 29–36. IEEE Computer Society, 2004.
Forrest Shull and Raimund L. Feldmann. Building theories from multiple evidence sources. In Forrest Shull, Janice Singer, and Dag I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 337–364. Springer, 2008.
Dag I. K. Sjøberg, Tore Dybå, Bente C. D. Anda, and Jo E. Hannay. Building theories in software engineering. In Forrest Shull, Janice Singer, and Dag I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 312–336. Springer, 2008.
Victor R. Basili. The role of experimentation in software engineering: Past, current, and future. In 18th International Conference on Software Engineering, pages 442–449, Washington, DC, USA, 1996. IEEE Computer Society.
Donald T. Campbell and Julian Stanley. Experimental and Quasi-Experimental Designs for Research. Houghton Mifflin Co., Boston, MA, USA, 1963.
Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. Selecting empirical methods for software engineering research. In Forrest Shull, Janice Singer, and Dag I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 285–311. Springer, 2008.
Natalia Juristo and Ana M. Moreno. Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Boston/Dordrecht/London, 2001.
Barbara A. Kitchenham, Tore Dybå, and Magne Jørgensen. Evidence-based software engineering. In 26th International Conference on Software Engineering, pages 273–281, Washington, DC, USA, 2004. IEEE Computer Society.
Barbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721–734, 2002.
Dewayne E. Perry, Adam A. Porter, and Lawrence G. Votta. Empirical studies of software engineering: A roadmap. In Future of Software Engineering, 2000, pages 345–355, New York, NY, USA, 2000. ACM.
Dag I. K. Sjøberg, Tore Dybå, and Magne Jørgensen. The future of empirical methods in software engineering research. In Future of Software Engineering, pages 358–378, Washington, DC, USA, 2007. IEEE Computer Society.
Walter F. Tichy, Paul Lukowicz, Lutz Prechelt, and Ernst A. Heinz. Experimental evaluation in computer science: A quantitative study. Journal of Systems and Software, 28(1):9–18, 1995.
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén. Experimental Software Engineering: An Introduction. Kluwer Academic Publishers, Boston/Dordrecht/London, 2000.
Research Reporting Methods
Andreas Jedlitschka and Dietmar Pfahl. Reporting guidelines for controlled experiments in software engineering. In International Symposium on Empirical Software Engineering, pages 95–104, November 2005.
Kai Petersen and Claes Wohlin. Context in industrial software engineering research. In 3rd International Symposium on Empirical Software Engineering and Measurement, pages 401–404, October 2009.
Per Runeson and Martin Höst. Guidelines for conducting and reporting case study research in software engineering. In Empirical Software Engineering: An International Journal, 14(2):131–164, 2009.
Klaas-Jan Stol and Muhammad Ali Babar. Reporting empirical research in open source software: The state of practice. In Cornelia Boldyreff, Kevin Crowston, Björn Lundell, and Anthony I. Wasserman, editors, Open Source Ecosystems: Diverse Communities Interacting, pages 156–169. Springer, 2009.
Thomas S. Kuhn. The Structure of Scientific Revolutions. The University of Chicago Press, Chicago, IL, USA, third edition, 1996.
Nancy G. Leveson. High-pressure steam engines and computer software. In 14th International Conference on Software Engineering, pages 2–14, New York, NY, USA, 1992. ACM.
Karl R. Popper. The Logic of Scientific Discovery. Reprint, Routledge, New York, NY, USA,  2000.
Samuel H. Scudder. Take this fish and look at it. In The Prentice Hall Guide for College Writers, pages 53–56. Allyn & Bacon, 2010.